Why do I blog?

I've been asked in different ways and phrases about my motivation in responding to forum posts and more recently about this blog. The simple answer is that I enjoy it. I like writing, helping, and troubleshooting.

Readers of my posts will quickly note the style in which I interact - I ask a lot of questions. This is in part to confirm my understanding of the situation, but also in part to get the reader thinking about the underlying issue. Many times if I don't have the answer these are questions I'd ask myself in the course of resolving the issue.

Whenever possible I like to interject thoughts on larger issues like MS-Exchange principles, messaging design and architecture, and general troubleshooting techniques. I am a teacher-wannabe and enjoy interacting with people who are interested in learning.

My own troubleshooting technique stems from the philosophy that if you understand the underlying structure, you can always figure out the answer even if you've never encountered anything exactly like it before. This actually comes from my old Calculus professor's response to the class whining about having to memorize all the different integral forms. I never did get to that level of understanding in Calculus, but the philosophy works for many other subjects and especially so with computers.

Relays - good or bad?

I see many questions about relaying. It seems much of the confusion stems from an incomplete or incorrect understanding of what it means. Hopefully this will clear things up a bit.

What is relaying?

For any given message, you have a sending system (originator) and a recipient system. Most of the time, the sending system communicates directly with the recipient system. So, if you were sending a message to me, your Exchange server would communicate directly with mine.

There are times and circumstances where direct communication is not possible or desired. These circumstances may require that an intermediate system field the message and pass it along. This is what a relay is. It is a neither the sending system nor the recipient system.

Many ISPs work this way where you have to send all your outbound mail to them to be sent out to other Internet domains. The ISP accepts messages only from its known customers (via IP address) and rejects messages from others. Another common example of a valid relay is a Spam Filter appliance.

Open relays

A big problem that we Messaging Admins face is that there are "open relays" out in the world. An Open Relay is when a system accepts messages from everyone and will forward them to anyone. Spammers love these system because they can hide behind open relays to mask the true originating system. The default configuration of Exchange 5.5 unfortunately was set up to be an open relay. Many mail systems were unknowningly left that way, creating a paradise for spammers. Relaying with Exchange 2000 by default was closed to all but authenticated users, and Exchange 2003 continues that same default configuration.

Relay settings in Exchange 2003

These are set in the properties of the Default SMTP Virtual Server, on the Access tab.
If you select "all except the list below" in the relay settings with a blank list, you are actually saying "forward messages from everywhere". In other words, you will have configured an open relay and raised the frustration level of legitimate Messaging Admins everywhere.

When you select "only the list below" with a non-blank list, you are saying "don't forward the message unless it comes from a system on the list.

A final important configuration note

If you clear the checkbox for "Allow all computers which successfully authenticate to relay regardless of the list above", then Exchange servers within your organization will not be able to send messages to one another.

Using Telnet to simulate server communication

The best place to run Telnet is on the server which sends out your SMTP traffic. This will show you the same information that your SMTP engine receives when communicating with an outside system. Telnet allows you to specify the port through which to communicate. SMTP is defined as TCP port 25.

Open a command prompt window. Determine the FQDN or the IP address. If you need to determine this information, you can use NSLookup if you know the SMTP domain name you are attempting to connect to. For more information about this, read this article.

At the prompt, type telnet fqdn 25

If the receiving server is accepting SMTP communications, it will respond with an acknowledgement message indicating it is ready to receive your transmission. The acknowledgement should also indicate if it understands SMTP or ESMTP.

Type ehlo testdomain.com

There are two established protocols, SMTP and ESMTP (Enhanced SMTP). If the receiving system only understands SMTP, you must begin with helo. If the receiving system understands ESMTP, you may begin with either helo or ehlo.

If you receive an OK message from the receiving mail system, proceed. If not, double check the protocol named in the response to the telnet command.

Type
mail from:exchange.admin@testdomain.com

This indicates the reply address. Some receiving systems will compare the parameter from the ehlo command, and the domain listed in the address on the mail from: command to the domain name returned when performing a reverse DNS (RDNS) lookup on the IP address from which the message is coming. It is a method to combat address spoofing and more reliably identify undesirable senders.

If you are testing communications to an outside messaging system, you may need to use your actual domain name to be allowed to continue.

Type
rcpt to:valid.user@receivingdomain.com

This indicates the recipient address. If receivingdomain.com is not a domain being fielded by the receiving system, and the system does not allow relaying to receivingdomain.com, an error code will be returned.

Type data

This begins the actual message. Optionally, From:, To:, and BCC: can be entered at this time (to be covered in a future article).

Type subject: Test message via Telnet

Type a blank line - this denotes the end of the subject and the beginning of the message body.

Type This is a test
Type Please reply if received
Type a blank line
Type a period (".") and press Enter - this marks the end of the message body. The receiving system will understand and return a prompt.
Type quit

This ends the Telnet session and you will be returned to the OS prompt.

If everything has gone well, the message will be on its way to the recipient address. Give it a minute and check. You now know how to manually create and send an SMTP message! This can be a great troubleshooting tool, as you will receive reponses and acknowledgements from the receiving system that can aid in diagnosing a communication problem.

Using NSLookup to determine an SMTP receiving system

Background
NSLookup is a great tool that comes with Windows that allows you to search DNS for information. &nbspIt is especially useful to troubleshoot particular issues with Exchange. &nbspExchange is reliant upon DNS to know where to send outbound messages. &nbspWhen Exchange has problems getting messages to a particular domain, it's time to open the toolbox.

The best place to run NSLookup is on the server which sends out your SMTP traffic. &nbspThis will show you the same information that your SMTP engine uses when determining where to send mail to a particular domain.

Open a command prompt window
At the prompt, type nslookup
Type the command set type=mx
Type the registered domain name (e.g. domain.com)

You will receive a response similar to:

Non-authoritative answer:
domain.com MX preference = 10, mail exchanger = mail1.domain.com
domain.com MX preference = 20, mail exchanger = mail2.domain.com
domain.com MX preference = 30, mail exchanger = mail3.domain.com

Interpreting the NSLookup results
Your SMTP engine will attempt to use the MX records in ascending order according to their value. &nbspThe name associated with the MX record is what your engine will use. &nbspYou can simulate what the engine does by using the Telnet command. &nbspIn other words, the FQDN associated with the lowest numbered MX value would be the one that your SMTP engine would attempt to connect with.

Using the NSLookup results to test connectivity
In the simulated response shown above, you can test the readiness for receiving SMTP communications by using the Telnet command. &nbspIn a command-prompt window, type telnet mail1.domain.com 25. &nbspIf the system connected to the FQDN is accepting SMTP communications, you’ll receive a response.

DNS Root Servers

What are the root servers? These are the DNS servers from which all others get information.

Why would anyone care? To verify the information other mail systems see when trying to reach your SMTP domain.

How do I use the information? You can force NSLookup to poll a particular DNS server by using the command:
server ip_address or server fqdn

As an example, open a command prompt window
At the prompt, type nslookup
Type the command set type = mx
Type the command server a.root-servers.net
Type the registered domain name (e.g. domain.com)

You have requested the MX information for domain.com directly from one of the DNS root servers.

Here is IP information for the thirteen (13) root servers.
a.root-servers.net   198.41.0.4
b.root-servers.net   192.228.79.201
c.root-servers.net   192.33.4.12
d.root-servers.net   128.8.10.90
e.root-servers.net   192.203.230.10
f.root-servers.net   192.5.5.241
g.root-servers.net   192.112.36.4
h.root-servers.net   128.63.2.53
i.root-servers.net   192.36.148.17
j.root-servers.net   192.58.128.30
k.root-servers.net   193.0.14.129
l.root-servers.net   198.32.64.12
m.root-servers.net   202.12.27.33

The complete information can be found at the website: http://www.root-servers.org

Missing messages - Part 4

The message was delivered to the mailbox - where did it go?

This is the most common scenario [I mean, speaking as one Exchange Admin to another - what else could it be? ;) ].  As a personal aside, your goal is to figure out what happened and calmly point it out to the user.  The user will likely feel embarrassed already - no need to editorialize or lecture.

As discussed in Part 1, a successful message delivery typically means one of the following:

-  It reached the mailbox and was segregated or deleted by a system function

-  It reached the mailbox and was segregated or deleted by a client function

-  It reached the mailbox and was segregated or deleted by a user function

-  It reached the mailbox and was manually segregated or deleted

System function typically means forwarding configured in AD

To check for forwarding, open the Users & Computers console (ADUC) and open the properties of the recipient's object.  On the Exchange General tab go to Delivery Options.  Any forwarding configured at the Active Directory level will appear there.

Client functions include anti-virus/anti-spam filtering, and directing new messages to a personal folder

Check the console and logs of any 3rd-party anti-virus and anti-spam software.  Check the Junk E-mail folder in the user's mailbox.

Check all workstations this user logs on to for a profile that directs all new messages to a Personal Folder instead of to the mailbox.

User functions include rules, auto-archiving, and viewing filters

Check for and disable any viewing filters in Outlook (View-->ArrangeBy-->Custom)

Check for auto-archiving (File-->Archive), look in all Personal folders listed in the Outlook profile.  Search for all PST files on the local drive and all mapped drives.

Check for rules (Tools-->Rules and Alerts)

If the ruleset is empty, there is still a possibility that something formerly in rules is still acting on messages.  To make sure, close Outlook, then launch it again from a command line using the /cleanrules switch (e.g. outlook.exe /cleanrules)

If the ruleset is not empty and you wish to keep them, you can export the set to a file then import again later.

Remember that the Out Of Office function can also have rules.  If OOO is enabled, make sure you check that configuration for rules.

Manual processes initiated by the user

Look for and search any PST files in the Outlook profile and on the local drive.

Look in the Deleted Items folder.  Look at the Recover Deleted Items area.

Search the other folders for items which were Shift-Deleted.

Missing messages - Part 3

Message Tracking sees the message, but it was not delivered to the mailbox

If Message Tracking (MT) has a record of the messsage in question, Exchange has received it.  If it does not reach the mailbox, the message is typically:
-  stuck in a queue
-  stuck in a routing loop
-  segregated by anti-virus/anti-spam filtering

You can often gain insight as to what is happening by reading through the audit trail of the message in MT.

Search the local and inter-server queues on all your servers.  If found, try manually releasing it and see what happens.

Check the logs of any 3rd-party anti-virus and anti-spam software.

Missing messages - Part 2

Message Tracking does not find the message I was expecting, where could it be?

This situation calls for additional sleuthing.  You need to understand your messaging environment and all the systems a message passes through on its way to the Exchange server.  Identify each and check any available logs.  Configuration and operation of routers, firewalls, mail gateways, even managed layer-3 switches can have an effect on inbound mail.

How widespread is this issue?  Does it affect all inbound messages, a significant number of inbound messages, or a small number of inbound messages?  Look for any consistencies (sending domain, sending address, receiving address).

Test inbound routing by sending yourself a message from an outside mail system (e.g. Yahoo, Hotmail, gMail).  Test by sending the affected user a message from that same outside system.

This scenario can get very complicated and vary greatly from environment to environment because you are dealing with any number of different devices and configurations.  Take it systematically, start at the outside and work your way in. Test each step, raise logging levels if necessary.

Missing messages - Part 1

How to start

I have been approached many times by users claiming that they never received a particular Email message. So where does one start looking? Most of the following scenarios have happened to me. The rest are follow-up thoughts of my own.

I start by asking my user some questions:
- What was the sending address?
- Approximately what time was the message sent?
- Are you seeing other messages arrive?
- If necessary may I open your mailbox to investigate?

I could also ask if the sender received a "bounce" message (a.k.a. Non-Delivery Report, a.k.a. NDR), but that tends to take extra coordinative effort. It's easier to assume that the sender is fine and to search for issues in the environment you can control (i.e. your own). Prove your own system sound before trying to look for causes outside. Show that you want to solve problems and not look for someone to blame.

Armed with this information, let's consider some possibilities:
1. It never reached our systems
2. It reached our systems but did not reach Exchange
3. It reached Exchange but was not delivered to the recipient's mailbox
4. It reached the recipient's mailbox but does not appear in the client software

The list is sorted according to message flow, but that does not mean you have to investigate in the same order.

The first question I ask myself is, does Exchange think it was delivered to the recipient's mailbox? Most of the time I find that the message did in fact reach the recipient's mailbox and something was done to it either automatically or manually.

Use Message Tracking (MT) to confirm whether the message was delivered. Use the information obtained from the user as the search parameters. If MT finds the message (regardless of outcome), rule out #1 and #2. If MT reports "Message delivered locally to store", it reached the recipient ruling out #3.

At this point, let's break the investigation into three parts.

If you cannot find the message in MT, continue with Part 2, Message Tracking does not find the message I was expecting, where could it be?

If MT finds the message, but reports something other than "delivered locally", continue with Part 3, Message Tracking sees the message, but it was not delivered to the mailbox

If MT does not find the message, continue with Part 4, The message was delivered to the mailbox - where did it go?