Disappointing news from Microsoft
Link also located at:
http://news.com.com/Microsoft+pushes+spam-filtering+technology/2100-7355_3-5758365.html?tag=nefd.top?headline=Microsoft~pushes~spam-filtering~technology

Reports published today indicate that Microsoft may attempt to mark messages from domains without SenderID records as Spam for their MSN and Hotmail services sometime in the November timeframe in attempts to cut down on spam.

I tend to agree with some of those quoted in the article in that this seems to be an attempt to force SenderID to be adopted, and as such, may be doomed to fail. I hope this stance is re-visited before November.
- posted by Ben Winzenz @ 12:31 PM 0 comments

 
Image Lifesaver!

- posted by Ben Winzenz @ 10:16 AM 1 comments

 
Managing the TechEd Operations Network

- posted by Ben Winzenz @ 8:34 AM 0 comments

 
Fun food statistics from TechEd

Feeding the Masses What does it take to feed 13,000 people at Tech·Ed?

• 117,000 bottles of water
• 14,300 lattes
• 15,600 ice cream bars (it’s HOT out there)
• 4,000 pounds of chips and snack mix
• 2,300 dozen eggs
• 8,000 pounds of chicken
• 52,000 servings of juice

Yikes!  That's a lot of food!

- posted by Ben Winzenz @ 2:46 PM 0 comments

 
Exchange Security at Microsoft
Presented by Konstantin Ryvkin of Microsoft's internal Messaging Group.

Two key aspects that were identified.

1. Need to identify exactly what you are trying to protect
2. Need to identify what you are trying to protect it from

4 areas of security

1. Exchange environment security
2. Windows Server and Exchange Server security
3. Exchange communications security
4. Messaging Client Access Security

E-mail is more than just AV/AS

Statistic:
Month of December 04, of approx. 50,000,000 + message submission attempts to microsoft.com domain, only about 1,500,000 were legitimate!

Multi-layered defense is the key.
Combination of Connection Filtering, Sender and Recipient Filtering, and Intelligent Message Filtering implemented.

Exchange SMTP Gateways - Connection Filtering, Sender/Recipient Filtering, Anti-spam filtering
Exchange Hubs - Attachment Filtering Antivirus
Exchange Mailbox servers - no filtering taking place
Clients - Attachment blocking antivirus, Anti-spam

Two SMTP Virtual Servers approach for handling e-mail. Different SMTP servers handle inbound and outbound e-mail traffic. Makes gathering statistics/metrics much easier.

Connection Filtering:
RBL blocking - low overhead on the server itself.

Recipient Filtering:
NDR generation and delivery is expensive. Enabling "filter recipients not in the directory" rejects invalid recipients before the message payload is transmitted.

However, this can result in directory harvesting attack, so....
you implement tarpitting. Tarpitting delays responses to subsequent invalid recipients, so it slows down the potential attacker significantly. KB 842851 addresses how to implement this.

Implement Restrictions on who can send to sensitive or large distribution groups. Enable accepting messages only from "authenticated users" - this prevents anonymous sending to DL from internet. Also implement only accepting mail from certain groups/users.

To protect against spoofing:
Enable "resolve anonymous e-mail" under authentications settings (on SMTP Virtual Server)
Implement SPF/SenderID records.

To harden the Windows platform, implement Group Policies based on the role of the server.
i.e. set up different policies for Front-end serves, Gateway Servers, Mailbox Servers, Clustered Servers.
Server computer accounts are placed in appropriate OU's and GPO's applied to OU's. New servers have appropriate GPO applied based on role.

Securing Exchange data in transit:
HTTPS for access from the Internet to OWA (require SSL)
IPSec between all internal servers
RPC envcryption between server and clients
TLS encryption between Exchange and external SMTP Gateways.

Summary:
Remain current with software and update versions at all levels
Security at multiple levels – defense in depth
Establish layered e-mail hygiene defenses
Secure Exchange servers by role
Be cognizant of security for upgrade scenarios
Bring Exchange Front Server out of perimeter network. Use reverse proxy solutions for secure Exchange publishing (ISA).
Use only secure authentication methods. Enforce e-mail data encryption where needed.

This was a great session as an overview of what methods are available to help secure mail servers. More importantly, it portrayed how Microsoft uses the built-in features to achieve messaging security. I'd contend that Microsoft has a much more complex messaging infrastucture than many companies and that implementing many if not all of these would not take that much work for many companies.
- posted by Ben Winzenz @ 11:34 PM 0 comments

 
Anatomy of a Network Hack: How to get your network hacked in 10 easy steps

- posted by Ben Winzenz @ 3:59 PM 0 comments

 
Exchange Performance and Tuning - Paul Bowden
Paul's the guy that did lots of work on the Best Practices Analyzer, so this fits right in his area of expertise.

Storage Performance.

7 out of 10 times where there are performance issues, the cause is storage. Usually too few spindles to meet the demand (IOPS required exceed disk ability)

Exchange is I/O hungry.
1 database spindle for every 100 users.

Actions to take:

• Work out your I/O profile - Optimizing storage white paper. Typical profile of sending 30 messages per day, receiving 100 messages per day, 100mb mailbox size will result in approx. 0.75 IOPS.
• Tell your storage vendor how many IOPS you will produce
• set caching controllers to 100% write / 4KB cache page size

The old rule of "put your logs on dedicated RAID1, put your databases on dedicated RAID5" doesn't always hold true any more.

Read/write ratio is typically 75% read/ 25% write

Use DiskPar to align tracks (performance boost) (Windows 2000, Windows 2003 Resource Kit) - use this if you are using DAS. Some more advanced storage (e.g. SAN's) you may not have to worry about this.

Typical tools to check performance are Loadsim and Jetstress. Jetstress is much better to validate storage architecture. You give it the variables and let it go and it will hammer the storage system.

A common bottleneck can be the Host Bus Adapter (HBA)

Warning signs of disk bottlenecks would be:
Average Disk Latency is higher than 20ms

Perfmon - look at the Logical Disk counters instead of the Physical disk counters.

Fast Access to Active Directory

• If Exchange is slow accessing the AD directory service, store performance suffers and mail delivery is slow.
• Duble chec that the Directory Service Access (DSAccess) list only contains local domain controller or global catalog servers.
• Remember that round robin is used
• Round trip times shiould be less than 10ms.

Rule of thumb is that you should have 1 GC processor for every 4 Exchange processors.

Verify that the /3gb switch is being used on GC's with 1gb or more of memory - this was news to me.
Increase JET cache from 512mb to 1gb

• More of the Active Directory is cached in memory

*In case there is any confusion about the /3gb switch, it can be used on any Windows 2000 and 2003 systems with the exception of Windows 2000 Standard.

Correct use of /3gb switch
Mailbox and PF servers
when there is 1gb or more of RAM
Windows 2003 - implement /USERVA=3030

FE and Bridgehead servers
Do NOT implement regardless of memory!
Instead, you can tune the Windows memory settings. By default, XP/2003 will set memory usage for System cache. Installing Exchange will switch that over to Programs. On FE and Bridgehead/Routing servers, it's actually beneficial to switch that back to System Cache. This is because most of what a FE/Bridgehead will be doing is queuing mail, etc.

Using the /USERVA=3030 - reducing user mode memory and re-claiming 42mb. Allows for more System PTE's and makes kernel mode memory much happier.

HeapDecommitFreeBlockThreshold
Should be set toe 0x40000 on ALL Exchange servers.
By default, this is set to 0. This can cause fragmented bits of the process. Changng this setting tells the OS that it should only return virtual memory when it is in a much larger block.

PAE and /3GB.

PAE allows access to memory beyond 4gb
Hardware-enforced Data Execution Prevention

Three ways to enable PAE
/PAE switch in the boot.ini
Automatic enabling on newer machines with Windows 2003 and hot-add memory
Enable DEP in BIOS

Note: Exchange will NOT take advantage of memory beyond 4gb, so you need to be careful when enabling PAE. It puts additional pressure on the kernel, especially Free PTE's (System Page Table Entries). Running out of PTE's can cause serious problems, such as server dropping off network, or one of the HBA's suddenly disappearing.

Enabling Hyper-threading - DO IT!
setting in BIOS
Sometimes defaults to OFF
Can improve Exchange processor efficiency by 25%
Works with Windows 2000 and Windows 2003.

Deploy Exchange 2003 Service pack 1

• 30% reduction in I/O pattern. SP2 will be reducing I/O even more!
• Drastic reduction in log file replay times
• From 30 seconds to less than 2 seconds
• Support for 8-way hyper-threading servers
• Sets MaxDSNSize to 10mb by default - by default, there is no limit. This could cause issues where your server is bogged down with NDR's. Setting this registry value will still allow Exchange to send the DSN, but if the original message is more than 10mb, it will not be attached to the NDR.
• Detection of single-bit flip errors (ECC correction within databasse)

Hardware Device Drivers

• Disable unused hardware devices
• Especially consider network adapters and storage cards that are not being used.
• Use identical hardware in the server - results in one driver for multiple devices
• Be especially careful with graphics adapters!
• can result in serious PTE reduction
• keep drivers up to date
• switch to vga.sys driver - don't need blazing graphics on servers.

Setting the /3gb switch.
119,000 - goes down to 14,000 PTE's
Warning zone - if you get down to ~5,000 PTE's
Danger zone - if you get down to ~3,000 PTE's
If you run out of System PTE's, it will likely result in a Blue-screen.

Windows 2003 - can use /USERVA=3030. Windows 2000, use SystemPages registry parameter (steals from the Paged Pool)

TEMP Conversions

• For POP3, IMAP, and dedicated PF servers, redirect TEMP path to a dedicated fast disk.
• Stand-alone = System TEMP path
• Cluster = Cluster account TEMP path

Tune Antivirus scanners
Get latest avice from your antivirus vendor on optimum configuration for your server.

• Number of scanning threads
• memory/disk file size threshold

Remember to keep your antivirus binaries up-to-date

Avoids problems such as

• Virtual memory fragmentation
• non-paged pool errors
• service failures

Online Maintenance Schedule

• Skew online maintenance for large mailbox servers so there is at least a 15 minute gap between start times.
• Skew online maintenance and online backup schedule so that here is zero overlap
• Backup will kill maintenance threads if they collide
• Always make sure that online maintenance gets a chance to run
• NEVER disable online maintenance.

Log Buffer tuning
Exchange server 2003 defauts to 500 buffers (at least 512bytes)

• If disk latencies are reasonable, but log stalls are high, increase log buffer value in Active Directory
• This is especially critical for deployments that use replicated storage
• Use a multiple of 64 and never more than 9000

Mailbox Store Cache

• Store has a two hour cache for re-reading mailbox configuration settings
• Mailbox limits and thresholds
• Use Mailbox Cache Age Limit registry parameter to reduce wait time. (or you can restart IS) :-) Recommended to set this to perhaps 15 minutes.

Use Dedicated Routing servers

• Queueing messages use paged pool memory and increases CPU utilization
• Ensure that large mailbox servers can immediately offload messages to a local dedicated routing server
• For routing servers, split SMTP queue onto dedicated fast disk
• only implement CheckConnectorRestrictions registry parameter on routing servers

Reduce LinkState size

• Avoid creating or removing routing groups
• Use REMoniotr - inject to nullify old data if necessary (PSS tool)

Avoid 7-bit conversions
Pairs of Exchange Server 2003 bridgeheads on native mode orgs perform 8-bit MIME transfers to each other over RGC's.
All other combinations will force each message to go through 7-bit conversion, resulting in ~30% conversion overhead.

Implement Message Size Limits
Understand Client-add ins - most will increase the load on the server.
VAR solutions such as BES usually require store-level tuning.

Bottom line here is that there were actually a number of suggestions that I hadn't seen before. Paul always presents great sessions, and this was no exception.

- posted by Ben Winzenz @ 11:10 AM 4 comments

 
Paul Flessner's keynote

• Integrated development and debugging experience
• Execution location and programming language choice

• Asynchronous queuing for highly available applications
• Reliable messaging to scale out

• High performance ASP.NET 2.0 apps

•     Native XML Support in the DB

• Highest developer productivity for integration
• web services orchestration, Business process and Rules
• Scalable and available

• Simplified setup
• One click application development

•     Development experience designed for Web developers
•     Dramatic code reduction; common scenarios built-in
•     significant performance improvements; CacheSync, 64-bit

•     Ease of the web, power of windows
•     ClickOnce: easy to install and update
•     Online, offline & mobile devices

- posted by Ben Winzenz @ 9:28 AM 0 comments

 
TechEd end of Day One
Today didn't involve attending any (Public) sessions. I spent a few hours this morning staffing the Exchange cabana and spent all afternoon learning cool things about the upcoming changes in the next version of Exchange (codename E12). Unfortunately, I'm under an NDA for that, so I can't really share any of the cool info that we got. I will say that it appears Microsoft will provide a good compelling reason to upgrade to E12 when it is available. There are other public details already available, such as the splitting of servers into "roles" and such, but there will be lots of other goodies available in E12. I'm off to a dinner for the Exchange MVP's tonight.
- posted by Ben Winzenz @ 4:55 PM 0 comments

 
Awesome Exchange news!
Finally the feedback has been implemented.
http://www.microsoft.com/exchange/downloads/2003/sp2/faq.mspx

The great news here is that the much loathed 16gb limit in Exchange Standard edition (as well as Small Business Server) has finally been lifted. Standard edition will still be limited to 1 Mailbox Store, but the new limit will be 75gb. This will be awesome news to those customers that aren't big enough to need Enterprise edition, but are still running into issues with the 16gb limit. The upgrade is supposed to be painless. Once you install SP2, the store should automatically be upgraded.
- posted by Ben Winzenz @ 10:21 AM 1 comments

 
At the Exchange Cabana
Toughest question so far:

How to restrict a specific group of users so that they can only e-mail certain people externally.

There doesn't seem to be an elegant solution for this currently. The 2 thoughts were:
1. Use an SMTP connector and implement restrictions on the connector. The problem is that SMTP connectors are not granular enough to apply to e-mail addresses - they only go to the SMTP domain level.
2. Create mailboxes for the external contacts/vendors and then set up an Outlook rule to automatically forward messages to the external e-mail address. The problem with this solution is twofold. First, it forces you to enable Automatic forwarding to the internet. Second, any replies to messages forwarded would go back to the mailbox, essentially forcing you to monitor that new mailbox.

It apppears the Exchange team will be taking this back to see if someting like this can be put into Exchange 12.
- posted by Ben Winzenz @ 8:54 AM 0 comments

 
Another Techie Show break-in during the keynote

- posted by Ben Winzenz @ 8:29 AM 0 comments

 
Keynote

- posted by Ben Winzenz @ 8:22 AM 0 comments

 
Whew - wirelesss available lots of places

- posted by Ben Winzenz @ 8:11 AM 0 comments

 
TechEd 2005 First day

- posted by Ben Winzenz @ 7:48 AM 0 comments