A Collection of Random Thoughts
Thursday, June 23, 2005
Disappointing news from Microsoft
Link also located at:
Reports published today indicate that Microsoft may attempt to mark messages from domains without SenderID records as Spam for their MSN and Hotmail services sometime in the November timeframe in attempts to cut down on spam.
I tend to agree with some of those quoted in the article in that this seems to be an attempt to force SenderID to be adopted, and as such, may be doomed to fail. I hope this stance is re-visited before November.
Monday, June 20, 2005
This weekend, I had a SD card get corrupted. I'm guessing that it was corrupted because when I powered my digital camera on, I received a message stating that the card wasn't formatted and would I like to format it. The problem: There were lots of images that were on the card that hadn't been offloaded yet. The result of this revelation: an upset wife (many of the pictures were from our daughter's birthday party). Thus began the search for a savior. Google was my friend here. Within minutes, I had found a freeware application (actually several) that would recover images from SD cards (among others). I ended up choosing an application called Zero Assumption Digital Image Recovery to recover the images. Luckily, my laptop at home (HP Pavilion ZD7000) has a built-in card reader (my other card reader was at work), so I simply plugged in the SD card, then ran the application and told it to recover the images. I assume that flash media work much the same way as hard drives in that when you format a flash card (or hard drive), it doesn't actually remove the data, just makes it inaccessible. Anyways, my point here is that there weren't a ton of images on the card (probably 40 or 50 since the last formatting - it's a 256mb card and my camera is a 4.1MP), but the program recovered ~120 images. That was fine with me, though, as long as the important ones were there, which they were. There were only a few images of the ~120 that weren't completely recovered, which is pretty amazing.
Kudos to Zero Assumption Recovery for making this tool free. The information on their website states the following:
"Zero Assumption Digital Image Recovery is freeware with no functional limitations."
In addition to the Digital Image Recovery program, they also have programs for recovering files from DOS and Windows. I haven't looked at those, but if they are anything like the image recovery program, they will be pretty decent.
Friday, June 10, 2005
Managing the TechEd Operations Network
Wireless issues. It presented several challenges this year.
Registered Attendees: ~13,000
Users of scheduling tool: ~7,000
Technical infrastructure staff: 10
Wired/Wireless users: 7,500
Breakout sessions: 440
Hands-on labs: 140
Network uptime to date: 99.998%*
*network minus access through WAP's
Requirements for event network:
Create an "infrastructure in a box" that can be re-used across events
Allow the entire infrastructure to be managed from a single point
Allow the infrastructure to be "plugged in" to the venue's in-house network as easily as possible
Be able to set up and fully configure the infrastructure in under 5 days
Allow the wireless and exhibitor networks outbound internet access using at least the following ports:
http/https/smtp/pop3/pptp/gre/l2tp/ipsec/vpn/messenger/rdp...machines on this network are not managed by Teched personnel
CommNet network has unrestricted Internet access and unrestricted access to the TechEd events servers (fully managed by the TechEd staff)
CommNet users can access the TechEd
support at least 8000 concurrent wireless users
support up to 800 HOL
Support up to 700 CommNet PC's
Provide e-mail ad show web services for attendees
32 total servers
ISA 2004 as firewall
MOM 2005 to monitor network.
SMS to manage PC's/Servers
All in all, it was an interesting overview of how much it takes to set up the TechEd network infrastructure, but also how well it works together.
Thursday, June 09, 2005
Fun food statistics from TechEd
There are approx. 13,000 attendees at TechEd this year. These fun food stats were pulled from the TechEd home page.
Feeding the Masses What does it take to feed 13,000 people at Tech·Ed?
- 117,000 bottles of water
- 14,300 lattes
- 15,600 ice cream bars (it’s HOT out there)
- 4,000 pounds of chips and snack mix
- 2,300 dozen eggs
- 8,000 pounds of chicken
- 52,000 servings of juice
Yikes! That's a lot of food!
Wednesday, June 08, 2005
Exchange Security at Microsoft
Presented by Konstantin Ryvkin of Microsoft's internal Messaging Group.
Two key aspects that were identified.
1. Need to identify exactly what you are trying to protect
2. Need to identify what you are trying to protect it from
4 areas of security
1. Exchange environment security
2. Windows Server and Exchange Server security
3. Exchange communications security
4. Messaging Client Access Security
E-mail is more than just AV/AS
Month of December 04, of approx. 50,000,000 + message submission attempts to microsoft.com domain, only about 1,500,000 were legitimate!
Multi-layered defense is the key.
Combination of Connection Filtering, Sender and Recipient Filtering, and Intelligent Message Filtering implemented.
Exchange SMTP Gateways - Connection Filtering, Sender/Recipient Filtering, Anti-spam filtering
Exchange Hubs - Attachment Filtering Antivirus
Exchange Mailbox servers - no filtering taking place
Clients - Attachment blocking antivirus, Anti-spam
Two SMTP Virtual Servers approach for handling e-mail. Different SMTP servers handle inbound and outbound e-mail traffic. Makes gathering statistics/metrics much easier.
RBL blocking - low overhead on the server itself.
NDR generation and delivery is expensive. Enabling "filter recipients not in the directory" rejects invalid recipients before the message payload is transmitted.
However, this can result in directory harvesting attack, so....
you implement tarpitting. Tarpitting delays responses to subsequent invalid recipients, so it slows down the potential attacker significantly. KB 842851 addresses how to implement this.
Implement Restrictions on who can send to sensitive or large distribution groups. Enable accepting messages only from "authenticated users" - this prevents anonymous sending to DL from internet. Also implement only accepting mail from certain groups/users.
To protect against spoofing:
Enable "resolve anonymous e-mail" under authentications settings (on SMTP Virtual Server)
Implement SPF/SenderID records.
To harden the Windows platform, implement Group Policies based on the role of the server.
i.e. set up different policies for Front-end serves, Gateway Servers, Mailbox Servers, Clustered Servers.
Server computer accounts are placed in appropriate OU's and GPO's applied to OU's. New servers have appropriate GPO applied based on role.
Securing Exchange data in transit:
HTTPS for access from the Internet to OWA (require SSL)
IPSec between all internal servers
RPC envcryption between server and clients
TLS encryption between Exchange and external SMTP Gateways.
Remain current with software and update versions at all levels
Security at multiple levels – defense in depth
Establish layered e-mail hygiene defenses
Secure Exchange servers by role
Be cognizant of security for upgrade scenarios
Bring Exchange Front Server out of perimeter network. Use reverse proxy solutions for secure Exchange publishing (ISA).
Use only secure authentication methods. Enforce e-mail data encryption where needed.
This was a great session as an overview of what methods are available to help secure mail servers. More importantly, it portrayed how Microsoft uses the built-in features to achieve messaging security. I'd contend that Microsoft has a much more complex messaging infrastucture than many companies and that implementing many if not all of these would not take that much work for many companies.
Tuesday, June 07, 2005
Anatomy of a Network Hack: How to get your network hacked in 10 easy steps
This session was presented by Jesper Johansson, Senior Program Manager in the Security Tools department.
Need to figure out the things that really good hackers do
Is anyone after you???
Picture of construction road sign that shows
Question: If your company does direct deposit for payroll, how easy would it be to find out if there are people on your payroll that don't work for your company?
Example from last TechEd where an attendee went back and had his company do manual payroll for 1 cycle. They found "several" people on the payroll that were not employees.
Know your adversary!!!
Picture of a homeless man with a sign that reads "Will Code HTML for Food"
You need the right weapons!!!
Picture of a man strapped underneath an airplane with a machine gun
1. Try and ping the target
There really is no good reason to enable ICMP from outside to inside your network.
2. Portscan - this tells you what ports are open on the target machine.
Knocking down the side door...
The problem is not the database back-end - it is usually the Web App that is broken
Have you logged on with a Domain Admin account on a non-Domain Controller recently?
If you have, you have just degraded the security of your network to the least secure machine that you have logged on with that account.
Do you have identical accounts with the same password in different domains or forests?
Cracking passwords is a waste of time. If a hacker can get a list of the hashes, they won't waste time with cracking passwords.
Unrestricted/unfiltered internal traffic
Moral of the story:
Most networks today are built like eggshells. On the outside, the are hard, but on the inside, they are soft and gooey. All you have to do is find 1 hole through the outside, and typically you will then own the entire network.
1. Update resume
2. Hope the hacker does a good job running the network
3. Nuke and rebuild.
How to get your network hacked in 10 easy steps
1. Don't patch anything.
2. Run unhardened applications
3. Use one admin account, everywhere (you should be using different admin accounts for every machine)
4. Open lots of holes in your firewall
5. Allow unrestricted internal traffic
6. Allow all outbound traffic
7. Don't harden servers
8. Re-use your passwords
9. Use high-level service accounts in multiple places
10. Assume everything is OK
What they don't want you to do
1. Ensure everything is properly patched
2. Use properly hardened papplications
3. Use least priveleged
4.Open only necessary holes in firewalls
5. Restrict internal traffic
6. Restrict outbound traffic
7. Harden servers
8. Use unique pass phrasess or smart cards
9. Micro-manager service accoutns
10. Maintain a health level of paranoia
Book - Protect your Windows Network - Jesper Johansson and Steve Riley
Promo code JJSR6437
Exchange Performance and Tuning - Paul Bowden
Paul's the guy that did lots of work on the Best Practices Analyzer, so this fits right in his area of expertise.
7 out of 10 times where there are performance issues, the cause is storage. Usually too few spindles to meet the demand (IOPS required exceed disk ability)
Exchange is I/O hungry.
1 database spindle for every 100 users.
Actions to take:
- Work out your I/O profile - Optimizing storage white paper. Typical profile of sending 30 messages per day, receiving 100 messages per day, 100mb mailbox size will result in approx. 0.75 IOPS.
- Tell your storage vendor how many IOPS you will produce
- set caching controllers to 100% write / 4KB cache page size
The old rule of "put your logs on dedicated RAID1, put your databases on dedicated RAID5" doesn't always hold true any more.
Read/write ratio is typically 75% read/ 25% write
Use DiskPar to align tracks (performance boost) (Windows 2000, Windows 2003 Resource Kit) - use this if you are using DAS. Some more advanced storage (e.g. SAN's) you may not have to worry about this.
Typical tools to check performance are Loadsim and Jetstress. Jetstress is much better to validate storage architecture. You give it the variables and let it go and it will hammer the storage system.
A common bottleneck can be the Host Bus Adapter (HBA)
Warning signs of disk bottlenecks would be:
Average Disk Latency is higher than 20ms
Perfmon - look at the Logical Disk counters instead of the Physical disk counters.
Fast Access to Active Directory
- If Exchange is slow accessing the AD directory service, store performance suffers and mail delivery is slow.
- Duble chec that the Directory Service Access (DSAccess) list only contains local domain controller or global catalog servers.
- Remember that round robin is used
- Round trip times shiould be less than 10ms.
Verify that the /3gb switch is being used on GC's with 1gb or more of memory - this was news to me.
Increase JET cache from 512mb to 1gb
- More of the Active Directory is cached in memory
Correct use of /3gb switch
Mailbox and PF servers
when there is 1gb or more of RAM
Windows 2003 - implement /USERVA=3030
FE and Bridgehead servers
Do NOT implement regardless of memory!
Instead, you can tune the Windows memory settings. By default, XP/2003 will set memory usage for System cache. Installing Exchange will switch that over to Programs. On FE and Bridgehead/Routing servers, it's actually beneficial to switch that back to System Cache. This is because most of what a FE/Bridgehead will be doing is queuing mail, etc.
Using the /USERVA=3030 - reducing user mode memory and re-claiming 42mb. Allows for more System PTE's and makes kernel mode memory much happier.
Should be set toe 0x40000 on ALL Exchange servers.
By default, this is set to 0. This can cause fragmented bits of the process. Changng this setting tells the OS that it should only return virtual memory when it is in a much larger block.
PAE and /3GB.
PAE allows access to memory beyond 4gb
Hardware-enforced Data Execution Prevention
Three ways to enable PAE
/PAE switch in the boot.ini
Automatic enabling on newer machines with Windows 2003 and hot-add memory
Enable DEP in BIOS
Note: Exchange will NOT take advantage of memory beyond 4gb, so you need to be careful when enabling PAE. It puts additional pressure on the kernel, especially Free PTE's (System Page Table Entries). Running out of PTE's can cause serious problems, such as server dropping off network, or one of the HBA's suddenly disappearing.
Enabling Hyper-threading - DO IT!
setting in BIOS
Sometimes defaults to OFF
Can improve Exchange processor efficiency by 25%
Works with Windows 2000 and Windows 2003.
Deploy Exchange 2003 Service pack 1
- 30% reduction in I/O pattern. SP2 will be reducing I/O even more!
- Drastic reduction in log file replay times
- From 30 seconds to less than 2 seconds
- Support for 8-way hyper-threading servers
- Sets MaxDSNSize to 10mb by default - by default, there is no limit. This could cause issues where your server is bogged down with NDR's. Setting this registry value will still allow Exchange to send the DSN, but if the original message is more than 10mb, it will not be attached to the NDR.
- Detection of single-bit flip errors (ECC correction within databasse)
- Disable unused hardware devices
- Especially consider network adapters and storage cards that are not being used.
- Use identical hardware in the server - results in one driver for multiple devices
- Be especially careful with graphics adapters!
- can result in serious PTE reduction
- keep drivers up to date
- switch to vga.sys driver - don't need blazing graphics on servers.
Setting the /3gb switch.
119,000 - goes down to 14,000 PTE's
Warning zone - if you get down to ~5,000 PTE's
Danger zone - if you get down to ~3,000 PTE's
If you run out of System PTE's, it will likely result in a Blue-screen.
Windows 2003 - can use /USERVA=3030. Windows 2000, use SystemPages registry parameter (steals from the Paged Pool)
- For POP3, IMAP, and dedicated PF servers, redirect TEMP path to a dedicated fast disk.
- Stand-alone = System TEMP path
- Cluster = Cluster account TEMP path
Get latest avice from your antivirus vendor on optimum configuration for your server.
- Number of scanning threads
- memory/disk file size threshold
Avoids problems such as
- Virtual memory fragmentation
- non-paged pool errors
- service failures
- Skew online maintenance for large mailbox servers so there is at least a 15 minute gap between start times.
- Skew online maintenance and online backup schedule so that here is zero overlap
- Backup will kill maintenance threads if they collide
- Always make sure that online maintenance gets a chance to run
- NEVER disable online maintenance.
Exchange server 2003 defauts to 500 buffers (at least 512bytes)
- If disk latencies are reasonable, but log stalls are high, increase log buffer value in Active Directory
- This is especially critical for deployments that use replicated storage
- Use a multiple of 64 and never more than 9000
Mailbox Store Cache
- Store has a two hour cache for re-reading mailbox configuration settings
- Mailbox limits and thresholds
- Use Mailbox Cache Age Limit registry parameter to reduce wait time. (or you can restart IS) :-) Recommended to set this to perhaps 15 minutes.
- Queueing messages use paged pool memory and increases CPU utilization
- Ensure that large mailbox servers can immediately offload messages to a local dedicated routing server
- For routing servers, split SMTP queue onto dedicated fast disk
- only implement CheckConnectorRestrictions registry parameter on routing servers
- Avoid creating or removing routing groups
- Use REMoniotr - inject to nullify old data if necessary (PSS tool)
Pairs of Exchange Server 2003 bridgeheads on native mode orgs perform 8-bit MIME transfers to each other over RGC's.
All other combinations will force each message to go through 7-bit conversion, resulting in ~30% conversion overhead.
Implement Message Size Limits
Understand Client-add ins - most will increase the load on the server.
VAR solutions such as BES usually require store-level tuning.
Bottom line here is that there were actually a number of suggestions that I hadn't seen before. Paul always presents great sessions, and this was no exception.
Paul Flessner's keynote
Getting connected is the main topic.
Gave some historical dates of different eras of communication, such as the first wired message, first telephone call, etc.
1994 - The Internet as we know it was released to the public by the government (of course, everyone knows that Al Gore created the Internet - HA!)
2004 - ~ 500 million people on the internet. Figure is projected to double by 2005 to ~ 1 billion.
Connected systems - making it easier
Visual Studio 2005
SQL Server 2005
BizTalk server 2006
Dev Ready - Database development
Integrated with Visual Studio and .NET
- Integrated development and debugging experience
- Execution location and programming language choice
SQL Server Service Broker
- Asynchronous queuing for highly available applications
- Reliable messaging to scale out
- High performance ASP.NET 2.0 apps
XML Data type
- Native XML Support in the DB
The Center of your connected Architecture
- Highest developer productivity for integration
- web services orchestration, Business process and Rules
- Scalable and available
New in BizTalk Server 2006
- Simplified setup
- One click application development
Team Development with Visual Studio
Infrastructure Architect -> Solution Architect -> Developer -> Tester -> Project Manager
Design for operations
Quality early & others
50 -75% code reduction for most scenarios
- Development experience designed for Web developers
- Dramatic code reduction; common scenarios built-in
- significant performance improvements; CacheSync, 64-bit
Smart client Development
- Ease of the web, power of windows
- ClickOnce: easy to install and update
- Online, offline & mobile devices
Visual Studio 2005 Demo
Some new changes coming with SQL 2005 include:
Smart online repair
Monday, June 06, 2005
TechEd end of Day One
Today didn't involve attending any (Public) sessions. I spent a few hours this morning staffing the Exchange cabana and spent all afternoon learning cool things about the upcoming changes in the next version of Exchange (codename E12). Unfortunately, I'm under an NDA for that, so I can't really share any of the cool info that we got. I will say that it appears Microsoft will provide a good compelling reason to upgrade to E12 when it is available. There are other public details already available, such as the splitting of servers into "roles" and such, but there will be lots of other goodies available in E12. I'm off to a dinner for the Exchange MVP's tonight.
Awesome Exchange news!
Finally the feedback has been implemented.
The great news here is that the much loathed 16gb limit in Exchange Standard edition (as well as Small Business Server) has finally been lifted. Standard edition will still be limited to 1 Mailbox Store, but the new limit will be 75gb. This will be awesome news to those customers that aren't big enough to need Enterprise edition, but are still running into issues with the 16gb limit. The upgrade is supposed to be painless. Once you install SP2, the store should automatically be upgraded.
At the Exchange Cabana
Toughest question so far:
How to restrict a specific group of users so that they can only e-mail certain people externally.
There doesn't seem to be an elegant solution for this currently. The 2 thoughts were:
1. Use an SMTP connector and implement restrictions on the connector. The problem is that SMTP connectors are not granular enough to apply to e-mail addresses - they only go to the SMTP domain level.
2. Create mailboxes for the external contacts/vendors and then set up an Outlook rule to automatically forward messages to the external e-mail address. The problem with this solution is twofold. First, it forces you to enable Automatic forwarding to the internet. Second, any replies to messages forwarded would go back to the mailbox, essentially forcing you to monitor that new mailbox.
It apppears the Exchange team will be taking this back to see if someting like this can be put into Exchange 12.
Another Techie Show break-in during the keynote
The new trend is Free- range information workers. Using Hot-spots to work from anywhere. They are the new "spoiled brats of business"
Top 5 requests from these "spoiled brats":
1. I just want 1 identity and password for my desktop and PDA and everything else. How hard can that be?
2. I had sending e-mails and IM's and no one gets back to me. I want to know where everyone is at all times!
3. I'm tired of bringing every laptop I use to IT. Even the laptop I use on my unsecured AP at home that my son uses to download mp3's, etc.
4. Why can't my calendar, e-mail, contacts be synchronized with any device that I use. Automatically. Laptop, cell phone, PDA, Smartphone, Smart refrigerator, stupid coffee maker, etc.
5. Self service website
Standard Response from IT Pros.... NO!
Not any more.
The keynote started off with a funny clip called "The Techie Show". It's a different way to open the keynote. A little bit of improv (a la Late Night TV). Funny announcements during the improv...
new Partnership between Toyota and Apple. New product will be a new Hybrid Prius Ipod. It will be called the hy_Pod. 78MPG and 78million songs.
Survivor IT - each week one person will be outsourced.
beta.google.oil - website to search for what part of your neighborhood will be drilled next...
Theme for his speech is "Enabling people". Their aim is to provide us with the tools needed to do our jobs successfully.
Whew - wirelesss available lots of places
For a minute, I thought I would have to hurt someone. As another attendee commented this morning, if you're at a Geek conference, there better dang well be wireless throughout the conference location. I'm up in the Overflow room (A 320 - third level) and I still have a very good connection to the wireless network. Kudos to the folks that set up the wireless network. The last experience I had with the wireless setup was in Dallas, and it wasn't good. I had the same bad experience at Exchange Connections last fall. In fact, I don't believe there was ANY wireless access at all. Note to the Connections organizers - make sure there is wireless. Attendees will REALLY appreciate it.
TechEd 2005 First day
I got into Orlando last night after spending the entire day in airports and on planes. Don't ask. Now that I'm finally at the conference, registration was quick and painless (as it always has been), and I was able to make it just in time for the pre-conference training session that was put on for the staff. The main point that they made was that as Staff, we are ALWAYS representing Microsoft and as such to be careful in what we say. They gave some other tips such as how to deal with media and folks such as that, and to only make comments about our area of expertise. Makes sense.
The Convention Center built a whole new annex a few years back, and that is where the conference is. Nice and open, and there appears to be plenty of space. I'm waiting to see how widespread the wireless network is - hopefully it will be available throughout the entire convention center. I'll have to view the Steve B. keynote from the overflow room, but it should be exciting. CommNet appears to be using the same old computers that have been used for the last 5 years, so we'll see how well they work out. Probably won't use them unless I need to access the internal conference network. Gotta head up for the keynote. I'll blog more later.