.comment-link {margin-left:.6em;}
A Collection of Random Thoughts
Tuesday, June 07, 2005
Exchange Performance and Tuning - Paul Bowden
Paul's the guy that did lots of work on the Best Practices Analyzer, so this fits right in his area of expertise.

Storage Performance.

7 out of 10 times where there are performance issues, the cause is storage. Usually too few spindles to meet the demand (IOPS required exceed disk ability)

Exchange is I/O hungry.
1 database spindle for every 100 users.

Actions to take:

The old rule of "put your logs on dedicated RAID1, put your databases on dedicated RAID5" doesn't always hold true any more.

Read/write ratio is typically 75% read/ 25% write

Use DiskPar to align tracks (performance boost) (Windows 2000, Windows 2003 Resource Kit) - use this if you are using DAS. Some more advanced storage (e.g. SAN's) you may not have to worry about this.

Typical tools to check performance are Loadsim and Jetstress. Jetstress is much better to validate storage architecture. You give it the variables and let it go and it will hammer the storage system.

A common bottleneck can be the Host Bus Adapter (HBA)

Warning signs of disk bottlenecks would be:
Average Disk Latency is higher than 20ms

Perfmon - look at the Logical Disk counters instead of the Physical disk counters.

Fast Access to Active Directory
Rule of thumb is that you should have 1 GC processor for every 4 Exchange processors.

Verify that the /3gb switch is being used on GC's with 1gb or more of memory - this was news to me.
Increase JET cache from 512mb to 1gb
*In case there is any confusion about the /3gb switch, it can be used on any Windows 2000 and 2003 systems with the exception of Windows 2000 Standard.

Correct use of /3gb switch
Mailbox and PF servers
when there is 1gb or more of RAM
Windows 2003 - implement /USERVA=3030

FE and Bridgehead servers
Do NOT implement regardless of memory!
Instead, you can tune the Windows memory settings. By default, XP/2003 will set memory usage for System cache. Installing Exchange will switch that over to Programs. On FE and Bridgehead/Routing servers, it's actually beneficial to switch that back to System Cache. This is because most of what a FE/Bridgehead will be doing is queuing mail, etc.

Using the /USERVA=3030 - reducing user mode memory and re-claiming 42mb. Allows for more System PTE's and makes kernel mode memory much happier.

Should be set toe 0x40000 on ALL Exchange servers.
By default, this is set to 0. This can cause fragmented bits of the process. Changng this setting tells the OS that it should only return virtual memory when it is in a much larger block.

PAE and /3GB.

PAE allows access to memory beyond 4gb
Hardware-enforced Data Execution Prevention

Three ways to enable PAE
/PAE switch in the boot.ini
Automatic enabling on newer machines with Windows 2003 and hot-add memory
Enable DEP in BIOS

Note: Exchange will NOT take advantage of memory beyond 4gb, so you need to be careful when enabling PAE. It puts additional pressure on the kernel, especially Free PTE's (System Page Table Entries). Running out of PTE's can cause serious problems, such as server dropping off network, or one of the HBA's suddenly disappearing.

Enabling Hyper-threading - DO IT!
setting in BIOS
Sometimes defaults to OFF
Can improve Exchange processor efficiency by 25%
Works with Windows 2000 and Windows 2003.

Deploy Exchange 2003 Service pack 1
Hardware Device Drivers

Setting the /3gb switch.
119,000 - goes down to 14,000 PTE's
Warning zone - if you get down to ~5,000 PTE's
Danger zone - if you get down to ~3,000 PTE's
If you run out of System PTE's, it will likely result in a Blue-screen.

Windows 2003 - can use /USERVA=3030. Windows 2000, use SystemPages registry parameter (steals from the Paged Pool)

TEMP Conversions
Tune Antivirus scanners
Get latest avice from your antivirus vendor on optimum configuration for your server.
Remember to keep your antivirus binaries up-to-date

Avoids problems such as
Online Maintenance Schedule
Log Buffer tuning
Exchange server 2003 defauts to 500 buffers (at least 512bytes)

Mailbox Store Cache
Use Dedicated Routing servers
Reduce LinkState size
Avoid 7-bit conversions
Pairs of Exchange Server 2003 bridgeheads on native mode orgs perform 8-bit MIME transfers to each other over RGC's.
All other combinations will force each message to go through 7-bit conversion, resulting in ~30% conversion overhead.

Implement Message Size Limits
Understand Client-add ins - most will increase the load on the server.
VAR solutions such as BES usually require store-level tuning.

Bottom line here is that there were actually a number of suggestions that I hadn't seen before. Paul always presents great sessions, and this was no exception.

Do NOT, I repeat DO NOT ever turn PAE on on an Exchange Server.

MS has KB on this at http://support.microsoft.com/?id=827281

I spent 6 months chasing down random crashes on my Exchange Servers and it turned out to be caused by PAE being enabled.
That KB article refers to not using the /nopae switch and states that the /nopae switch stops working after Windows 2003 SP1 is installed. In Paul's presentation, he actually said that having PAE enabled shouldn't cause problems. Sure, Exchange can't take advantage of the additional memory, but there are scenarios where PAE can be beneficial. For instance, as the presentation referred to, if you enable Data Execution Protection (DEP), that will enable PAE.

I just spoke with Paul. As of about a week ago, Microsoft now supports using PAE with the /3GB switch. However, this can get you into a precarious situation with not having much in terms of System PTE's free. To overcome this problem, you can tune the /USERVA= switch and lower the number. I believe the number now recommended with PAE enabled (not controlled by boot.ini) is USERVA=2970. This would give 60mb back to System VM instead of just 42mb with 3030 as the value. Hope that helps.
The webcast is coming soon!

Post a Comment

Links to this post:

Create a Link

<< Home

Powered by Blogger