Among the more common questions I get from my users concerns Non-Delivery Reports (NDRs) they receive for messages they didn't send. Some users are understandably upset because many of the subject lines would lead you to question their character.
In addition, I see lots of questions from messaging administrators concerned that their environment is an open relay because either they are receiving a lot of NDRs from outside systems, or their own queues are clogged with NDRs to other SMTP domains.
Recently I posted this response at MSExchange.org to an Exchange administrator searching for a reasonable explanation for the flood of inbound NDRs his users were receiving.
===============
Consider this scenario:
1. I am a "Secret Creator of Unwanted Messages" (a.k.a. SCUM)
2. There is an SMTP domain named company.com
3. The address no.one@company.com does not exist
4. Your SMTP domain is unsuspecting-user.com
5. You have a user with the address someone@unsuspecting-user.com
6. I send out a message addressed from someone@unsuspecting-user.com and to no.one@company.com
The message goes out from my secret lair in a nearby septic tank out to the Internet.
The message then gets delivered to the server at company.com. That server accepts the message, attempts to find a match for no.one@company.com and discovers there is no such address. Being the RFC-compliant system it is, it dutifully creates and sends out an NDR to the sender. The trouble is, it thinks the sender is someone@unsuspecting-user.com and so sends the NDR there.
===============
This isn't to say that we shouldn't watch for signs that our system has been compromised, but more on that later...
Tuesday, October 23, 2007
Thursday, August 30, 2007
GAL vs. OAB - Why can't I see my AD changes?
When you run Outlook in non-cache mode, or connect using Outlook Web Access (OWA), you access the Global Address List (GAL) directly. Any changes made will appear as soon as your AD forest is replicated.
When you run Outlook in cache mode, you are viewing an offline copy of the GAL called an offline address book (a.k.a. offline address list). By default, Exchange rebuilds the OAB once a day. Also by default, Outlook downloads the OAB once when you launch it - thinking it doesn't need to check more often because of the rebuild schedule.
If you need to force a new entry to show up immediately, you need to do two things. First, manually rebuild the OAB. Open the System Manager (ESM) and expand Recipients then click on Offline Address Lists. In the right pane right click Default Offline Address List and select Rebuild. You should then wait anywhere from 2 to 10 minutes (depending upon how many entries are in your GAL). Second, in Outlook (running in cache mode) go to Tools-->Send/Receive and select Download Address Book. You should then see the new entry.
If you wish to view or change the rebuild schedule, open the ESM and expand Recipients. Open the properties of the particular OAB in question. On the General tab, there is a field for Update interval. By default this is set to run daily - early in the morning. If you wish to have the OAB rebuild more than once a day, you can select "Use custom schedule" and create your schedule.
When you run Outlook in cache mode, you are viewing an offline copy of the GAL called an offline address book (a.k.a. offline address list). By default, Exchange rebuilds the OAB once a day. Also by default, Outlook downloads the OAB once when you launch it - thinking it doesn't need to check more often because of the rebuild schedule.
If you need to force a new entry to show up immediately, you need to do two things. First, manually rebuild the OAB. Open the System Manager (ESM) and expand Recipients then click on Offline Address Lists. In the right pane right click Default Offline Address List and select Rebuild. You should then wait anywhere from 2 to 10 minutes (depending upon how many entries are in your GAL). Second, in Outlook (running in cache mode) go to Tools-->Send/Receive and select Download Address Book. You should then see the new entry.
If you wish to view or change the rebuild schedule, open the ESM and expand Recipients. Open the properties of the particular OAB in question. On the General tab, there is a field for Update interval. By default this is set to run daily - early in the morning. If you wish to have the OAB rebuild more than once a day, you can select "Use custom schedule" and create your schedule.
Saturday, July 14, 2007
Lesson learned - Recovering from corruption
As if the previous lesson learned wasn't enough, it led to another painful lesson.
Our Microsoft PSS engineer had us take a backup of the current mailbox stores that wouldn't mount along with the transactional logs so that we'd have something to fall back on should things not go well. Our mailbox stores are large enough that the process would take several hours. Rather than have new mail bounce he had us create empty databases by configuring the file names different from the original files. This gave the new messages a place to go.
After the backup completed, we restored one mailbox store files back to the previous day (which took a couple of hours). We then went through several rounds of replaying the logs until we figured out where the corruption started. When that process finally completed, we had a restored mailbox store as good as we could get it, plus a newly created mailbox store with a couple of days worth of messages. The final step is to merge the two. This is done by making one of the stores the Recovery Storage Group and merging the new data with the old.
I'll interrupt the story here to say that because of the length of time involved with the process, we ended up doing this with two different groups of people. The first group completed the merge and all appeared well. The second group went through the same process with another mailbox store with a different PSS engineer.
The bad news
When the second group got to the merge, it was taking a very long time. Much longer than it took the first group. We could do nothing but watch the merge wizard slowly process each mailbox. Our mailbox store was large enough that the entire process took over a day to complete. Our users were patient and understanding, but at the same time displeased.
Lesson learned
Some of you may have recognized that what we were doing was performing a Dial-tone Restore. Henrik Walther wrote an excellent set of articles on this subject at MSExchange.org. The first group did exactly what Henrik described, and the second group had left out one important step which would have saved us many hours of frustration. Before performing the merge, you need to swap the mailbox stores so that you are merging the small into the large. The mailbox store was somewhere between 75GB and 100GB and therefore took a very long time!
I strongly urge everyone to read through Henrik's articles to familiarize yourself with the process. You never know when you'll need it!
Our Microsoft PSS engineer had us take a backup of the current mailbox stores that wouldn't mount along with the transactional logs so that we'd have something to fall back on should things not go well. Our mailbox stores are large enough that the process would take several hours. Rather than have new mail bounce he had us create empty databases by configuring the file names different from the original files. This gave the new messages a place to go.
After the backup completed, we restored one mailbox store files back to the previous day (which took a couple of hours). We then went through several rounds of replaying the logs until we figured out where the corruption started. When that process finally completed, we had a restored mailbox store as good as we could get it, plus a newly created mailbox store with a couple of days worth of messages. The final step is to merge the two. This is done by making one of the stores the Recovery Storage Group and merging the new data with the old.
I'll interrupt the story here to say that because of the length of time involved with the process, we ended up doing this with two different groups of people. The first group completed the merge and all appeared well. The second group went through the same process with another mailbox store with a different PSS engineer.
The bad news
When the second group got to the merge, it was taking a very long time. Much longer than it took the first group. We could do nothing but watch the merge wizard slowly process each mailbox. Our mailbox store was large enough that the entire process took over a day to complete. Our users were patient and understanding, but at the same time displeased.
Lesson learned
Some of you may have recognized that what we were doing was performing a Dial-tone Restore. Henrik Walther wrote an excellent set of articles on this subject at MSExchange.org. The first group did exactly what Henrik described, and the second group had left out one important step which would have saved us many hours of frustration. Before performing the merge, you need to swap the mailbox stores so that you are merging the small into the large. The mailbox store was somewhere between 75GB and 100GB and therefore took a very long time!
I strongly urge everyone to read through Henrik's articles to familiarize yourself with the process. You never know when you'll need it!
Friday, July 13, 2007
Addendum - Stopping Exchange services
Andy Grogan wrote an excellent commentary to my previous post, and I wrote in return. I don't know how many people would actually read the comments, and the points made are important enough that I wanted them to be more than a footnote. I left Andy's comments verbatim, and added a couple of notes to mine.
Andy wrote:
"The problem is, how do you know when the store is being written to? or not, I have seen a similar problem before CPU 100 % nothing much responding in terms of Exchange so you naturally try to stop the service. Then you get the dreaded "The Microsoft Information Store Service did not respond to the stop request in a timely fashion" and hang on stopping. This then raises - how long do you wait - and hour, two hours - a day?"
"I have in the past opened up Task Manager and added in the IO, and Bytes counters to see if the store process is writing, but its not fool proof. Personally I would love Microsoft to put some options into ending processes like there are in Unix where you have multiple levels to killing a process - sorry for the ramble - I just don't think there was much you could have done your were in a hole that most of us Exchange folks face at some point - do I pull the plug or don't I?"
The issues Andy raises in his comment are exactly the same questions and issues that haunted me. That's why I've adopted this new strategy which seems to work (at least it alleviates my fears). Dismount the stores first - don't try to do anything with the services. In fact, don't do anything with the services until the stores dismount.
In an emergency, I'd pull the plug on new messages (e.g. block port 25) and cause bounces rather than risk corruption. That will allow the stores to eventually catch up and quiet down. Then dismount the stores, then finally stop the services.
Andy wrote:
"The problem is, how do you know when the store is being written to? or not, I have seen a similar problem before CPU 100 % nothing much responding in terms of Exchange so you naturally try to stop the service. Then you get the dreaded "The Microsoft Information Store Service did not respond to the stop request in a timely fashion" and hang on stopping. This then raises - how long do you wait - and hour, two hours - a day?"
"I have in the past opened up Task Manager and added in the IO, and Bytes counters to see if the store process is writing, but its not fool proof. Personally I would love Microsoft to put some options into ending processes like there are in Unix where you have multiple levels to killing a process - sorry for the ramble - I just don't think there was much you could have done your were in a hole that most of us Exchange folks face at some point - do I pull the plug or don't I?"
The issues Andy raises in his comment are exactly the same questions and issues that haunted me. That's why I've adopted this new strategy which seems to work (at least it alleviates my fears). Dismount the stores first - don't try to do anything with the services. In fact, don't do anything with the services until the stores dismount.
In an emergency, I'd pull the plug on new messages (e.g. block port 25) and cause bounces rather than risk corruption. That will allow the stores to eventually catch up and quiet down. Then dismount the stores, then finally stop the services.
Wednesday, July 11, 2007
Lesson Learned - Stopping Exchange services
I inadvertently corrupted a couple of mailbox stores (like anyone actually causes corruption intentionally) because of a lack of understanding. I hope this story will save someone else some pain.
The story started when an Exchange server's CPU was running at 100% for a long time, causing my monitor to alert us. After some intial troubleshooting, it was decided to restart the Exchange services. The restart process failed, the CPU continued to run at 100%.
We waited an hour, and the status had not changed. Figuring a reboot would resolve any ailments, we did just that. The server took a long time, but finally rebooted after 15 minutes. When the server restarted, some of the mailbox stores did not mount. After spending some time to try to fix things, we put in a call to Microsoft PSS.
The bad news
We discovered that our actions corrupted several mailbox stores (this was on an Enterprise Edition server). In talking to Microsoft, we discovered that a shutdown or restart of the Operating System does not necessarily wait for all services to stop. The Information Store service apparently did not stop completely and after a timed delay, Windows shut itself down. We were told that the corruption happened because the store was actively being written to when the service stopped.
Lesson learned
With a new understanding of the Information Store service, whenever maintenance is performed on our Exchange servers, we always dismount all of the mailbox stores first. This assures that all "in flight" transactions are complete before the service is stopped.
The story started when an Exchange server's CPU was running at 100% for a long time, causing my monitor to alert us. After some intial troubleshooting, it was decided to restart the Exchange services. The restart process failed, the CPU continued to run at 100%.
We waited an hour, and the status had not changed. Figuring a reboot would resolve any ailments, we did just that. The server took a long time, but finally rebooted after 15 minutes. When the server restarted, some of the mailbox stores did not mount. After spending some time to try to fix things, we put in a call to Microsoft PSS.
The bad news
We discovered that our actions corrupted several mailbox stores (this was on an Enterprise Edition server). In talking to Microsoft, we discovered that a shutdown or restart of the Operating System does not necessarily wait for all services to stop. The Information Store service apparently did not stop completely and after a timed delay, Windows shut itself down. We were told that the corruption happened because the store was actively being written to when the service stopped.
Lesson learned
With a new understanding of the Information Store service, whenever maintenance is performed on our Exchange servers, we always dismount all of the mailbox stores first. This assures that all "in flight" transactions are complete before the service is stopped.
Monday, July 9, 2007
Microsoft MVP!
I got to spend a little time away from everything Exchange related (a.k.a. vacation) and returned to find a nice surprise amongst my Email. I have been named a Microsoft MVP! I will be spending the next few days seriously studying my privileges and responsibilities. Thanks to James Chong and everyone else who was involved in my obtaining this honor. I will do everything in my power to prove myself worthy of your support!
Wednesday, June 27, 2007
Why do I blog?
I've been asked in different ways and phrases about my motivation in responding to forum posts and more recently about this blog. The simple answer is that I enjoy it. I like writing, helping, and troubleshooting.
Readers of my posts will quickly note the style in which I interact - I ask a lot of questions. This is in part to confirm my understanding of the situation, but also in part to get the reader thinking about the underlying issue. Many times if I don't have the answer these are questions I'd ask myself in the course of resolving the issue.
Whenever possible I like to interject thoughts on larger issues like MS-Exchange principles, messaging design and architecture, and general troubleshooting techniques. I am a teacher-wannabe and enjoy interacting with people who are interested in learning.
My own troubleshooting technique stems from the philosophy that if you understand the underlying structure, you can always figure out the answer even if you've never encountered anything exactly like it before. This actually comes from my old Calculus professor's response to the class whining about having to memorize all the different integral forms. I never did get to that level of understanding in Calculus, but the philosophy works for many other subjects and especially so with computers.
Readers of my posts will quickly note the style in which I interact - I ask a lot of questions. This is in part to confirm my understanding of the situation, but also in part to get the reader thinking about the underlying issue. Many times if I don't have the answer these are questions I'd ask myself in the course of resolving the issue.
Whenever possible I like to interject thoughts on larger issues like MS-Exchange principles, messaging design and architecture, and general troubleshooting techniques. I am a teacher-wannabe and enjoy interacting with people who are interested in learning.
My own troubleshooting technique stems from the philosophy that if you understand the underlying structure, you can always figure out the answer even if you've never encountered anything exactly like it before. This actually comes from my old Calculus professor's response to the class whining about having to memorize all the different integral forms. I never did get to that level of understanding in Calculus, but the philosophy works for many other subjects and especially so with computers.
Tuesday, June 26, 2007
Relays - good or bad?
I see many questions about relaying. It seems much of the confusion stems from an incomplete or incorrect understanding of what it means. Hopefully this will clear things up a bit.
What is relaying?
For any given message, you have a sending system (originator) and a recipient system. Most of the time, the sending system communicates directly with the recipient system. So, if you were sending a message to me, your Exchange server would communicate directly with mine.
There are times and circumstances where direct communication is not possible or desired. These circumstances may require that an intermediate system field the message and pass it along. This is what a relay is. It is a neither the sending system nor the recipient system.
Many ISPs work this way where you have to send all your outbound mail to them to be sent out to other Internet domains. The ISP accepts messages only from its known customers (via IP address) and rejects messages from others. Another common example of a valid relay is a Spam Filter appliance.
Open relays
A big problem that we Messaging Admins face is that there are "open relays" out in the world. An Open Relay is when a system accepts messages from everyone and will forward them to anyone. Spammers love these system because they can hide behind open relays to mask the true originating system. The default configuration of Exchange 5.5 unfortunately was set up to be an open relay. Many mail systems were unknowningly left that way, creating a paradise for spammers. Relaying with Exchange 2000 by default was closed to all but authenticated users, and Exchange 2003 continues that same default configuration.
Relay settings in Exchange 2003
These are set in the properties of the Default SMTP Virtual Server, on the Access tab.
If you select "all except the list below" in the relay settings with a blank list, you are actually saying "forward messages from everywhere". In other words, you will have configured an open relay and raised the frustration level of legitimate Messaging Admins everywhere.
When you select "only the list below" with a non-blank list, you are saying "don't forward the message unless it comes from a system on the list.
A final important configuration note
If you clear the checkbox for "Allow all computers which successfully authenticate to relay regardless of the list above", then Exchange servers within your organization will not be able to send messages to one another.
What is relaying?
For any given message, you have a sending system (originator) and a recipient system. Most of the time, the sending system communicates directly with the recipient system. So, if you were sending a message to me, your Exchange server would communicate directly with mine.
There are times and circumstances where direct communication is not possible or desired. These circumstances may require that an intermediate system field the message and pass it along. This is what a relay is. It is a neither the sending system nor the recipient system.
Many ISPs work this way where you have to send all your outbound mail to them to be sent out to other Internet domains. The ISP accepts messages only from its known customers (via IP address) and rejects messages from others. Another common example of a valid relay is a Spam Filter appliance.
Open relays
A big problem that we Messaging Admins face is that there are "open relays" out in the world. An Open Relay is when a system accepts messages from everyone and will forward them to anyone. Spammers love these system because they can hide behind open relays to mask the true originating system. The default configuration of Exchange 5.5 unfortunately was set up to be an open relay. Many mail systems were unknowningly left that way, creating a paradise for spammers. Relaying with Exchange 2000 by default was closed to all but authenticated users, and Exchange 2003 continues that same default configuration.
Relay settings in Exchange 2003
These are set in the properties of the Default SMTP Virtual Server, on the Access tab.
If you select "all except the list below" in the relay settings with a blank list, you are actually saying "forward messages from everywhere". In other words, you will have configured an open relay and raised the frustration level of legitimate Messaging Admins everywhere.
When you select "only the list below" with a non-blank list, you are saying "don't forward the message unless it comes from a system on the list.
A final important configuration note
If you clear the checkbox for "Allow all computers which successfully authenticate to relay regardless of the list above", then Exchange servers within your organization will not be able to send messages to one another.
Saturday, June 9, 2007
Using Telnet to simulate server communication
The best place to run Telnet is on the server which sends out your SMTP traffic. This will show you the same information that your SMTP engine receives when communicating with an outside system. Telnet allows you to specify the port through which to communicate. SMTP is defined as TCP port 25.
Open a command prompt window. Determine the FQDN or the IP address. If you need to determine this information, you can use NSLookup if you know the SMTP domain name you are attempting to connect to. For more information about this, read this article.
At the prompt, type telnet fqdn 25
If the receiving server is accepting SMTP communications, it will respond with an acknowledgement message indicating it is ready to receive your transmission. The acknowledgement should also indicate if it understands SMTP or ESMTP.
Type ehlo testdomain.com
There are two established protocols, SMTP and ESMTP (Enhanced SMTP). If the receiving system only understands SMTP, you must begin with helo. If the receiving system understands ESMTP, you may begin with either helo or ehlo.
If you receive an OK message from the receiving mail system, proceed. If not, double check the protocol named in the response to the telnet command.
Type
mail from:exchange.admin@testdomain.com
This indicates the reply address. Some receiving systems will compare the parameter from the ehlo command, and the domain listed in the address on the mail from: command to the domain name returned when performing a reverse DNS (RDNS) lookup on the IP address from which the message is coming. It is a method to combat address spoofing and more reliably identify undesirable senders.
If you are testing communications to an outside messaging system, you may need to use your actual domain name to be allowed to continue.
Type
rcpt to:valid.user@receivingdomain.com
This indicates the recipient address. If receivingdomain.com is not a domain being fielded by the receiving system, and the system does not allow relaying to receivingdomain.com, an error code will be returned.
Type data
This begins the actual message. Optionally, From:, To:, and BCC: can be entered at this time (to be covered in a future article).
Type subject: Test message via Telnet
Type a blank line - this denotes the end of the subject and the beginning of the message body.
Type This is a test
Type Please reply if received
Type a blank line
Type a period (".") and press Enter - this marks the end of the message body. The receiving system will understand and return a prompt.
Type quit
This ends the Telnet session and you will be returned to the OS prompt.
If everything has gone well, the message will be on its way to the recipient address. Give it a minute and check. You now know how to manually create and send an SMTP message! This can be a great troubleshooting tool, as you will receive reponses and acknowledgements from the receiving system that can aid in diagnosing a communication problem.
Open a command prompt window. Determine the FQDN or the IP address. If you need to determine this information, you can use NSLookup if you know the SMTP domain name you are attempting to connect to. For more information about this, read this article.
At the prompt, type telnet fqdn 25
If the receiving server is accepting SMTP communications, it will respond with an acknowledgement message indicating it is ready to receive your transmission. The acknowledgement should also indicate if it understands SMTP or ESMTP.
Type ehlo testdomain.com
There are two established protocols, SMTP and ESMTP (Enhanced SMTP). If the receiving system only understands SMTP, you must begin with helo. If the receiving system understands ESMTP, you may begin with either helo or ehlo.
If you receive an OK message from the receiving mail system, proceed. If not, double check the protocol named in the response to the telnet command.
Type
mail from:exchange.admin@testdomain.com
This indicates the reply address. Some receiving systems will compare the parameter from the ehlo command, and the domain listed in the address on the mail from: command to the domain name returned when performing a reverse DNS (RDNS) lookup on the IP address from which the message is coming. It is a method to combat address spoofing and more reliably identify undesirable senders.
If you are testing communications to an outside messaging system, you may need to use your actual domain name to be allowed to continue.
Type
rcpt to:valid.user@receivingdomain.com
This indicates the recipient address. If receivingdomain.com is not a domain being fielded by the receiving system, and the system does not allow relaying to receivingdomain.com, an error code will be returned.
Type data
This begins the actual message. Optionally, From:, To:, and BCC: can be entered at this time (to be covered in a future article).
Type subject: Test message via Telnet
Type a blank line - this denotes the end of the subject and the beginning of the message body.
Type This is a test
Type Please reply if received
Type a blank line
Type a period (".") and press Enter - this marks the end of the message body. The receiving system will understand and return a prompt.
Type quit
This ends the Telnet session and you will be returned to the OS prompt.
If everything has gone well, the message will be on its way to the recipient address. Give it a minute and check. You now know how to manually create and send an SMTP message! This can be a great troubleshooting tool, as you will receive reponses and acknowledgements from the receiving system that can aid in diagnosing a communication problem.
Friday, June 8, 2007
Using NSLookup to determine an SMTP receiving system
Background
NSLookup is a great tool that comes with Windows that allows you to search DNS for information.  It is especially useful to troubleshoot particular issues with Exchange.  Exchange is reliant upon DNS to know where to send outbound messages.  When Exchange has problems getting messages to a particular domain, it's time to open the toolbox.
The best place to run NSLookup is on the server which sends out your SMTP traffic.  This will show you the same information that your SMTP engine uses when determining where to send mail to a particular domain.
Open a command prompt window
At the prompt, type nslookup
Type the command set type=mx
Type the registered domain name (e.g. domain.com)
You will receive a response similar to:
Non-authoritative answer:
domain.com MX preference = 10, mail exchanger = mail1.domain.com
domain.com MX preference = 20, mail exchanger = mail2.domain.com
domain.com MX preference = 30, mail exchanger = mail3.domain.com
Interpreting the NSLookup results
Your SMTP engine will attempt to use the MX records in ascending order according to their value.  The name associated with the MX record is what your engine will use.  You can simulate what the engine does by using the Telnet command.  In other words, the FQDN associated with the lowest numbered MX value would be the one that your SMTP engine would attempt to connect with.
Using the NSLookup results to test connectivity
In the simulated response shown above, you can test the readiness for receiving SMTP communications by using the Telnet command.  In a command-prompt window, type telnet mail1.domain.com 25.  If the system connected to the FQDN is accepting SMTP communications, you’ll receive a response.
NSLookup is a great tool that comes with Windows that allows you to search DNS for information.  It is especially useful to troubleshoot particular issues with Exchange.  Exchange is reliant upon DNS to know where to send outbound messages.  When Exchange has problems getting messages to a particular domain, it's time to open the toolbox.
The best place to run NSLookup is on the server which sends out your SMTP traffic.  This will show you the same information that your SMTP engine uses when determining where to send mail to a particular domain.
Open a command prompt window
At the prompt, type nslookup
Type the command set type=mx
Type the registered domain name (e.g. domain.com)
You will receive a response similar to:
Non-authoritative answer:
domain.com MX preference = 10, mail exchanger = mail1.domain.com
domain.com MX preference = 20, mail exchanger = mail2.domain.com
domain.com MX preference = 30, mail exchanger = mail3.domain.com
Interpreting the NSLookup results
Your SMTP engine will attempt to use the MX records in ascending order according to their value.  The name associated with the MX record is what your engine will use.  You can simulate what the engine does by using the Telnet command.  In other words, the FQDN associated with the lowest numbered MX value would be the one that your SMTP engine would attempt to connect with.
Using the NSLookup results to test connectivity
In the simulated response shown above, you can test the readiness for receiving SMTP communications by using the Telnet command.  In a command-prompt window, type telnet mail1.domain.com 25.  If the system connected to the FQDN is accepting SMTP communications, you’ll receive a response.
Thursday, June 7, 2007
DNS Root Servers
What are the root servers? These are the DNS servers from which all others get information.
Why would anyone care? To verify the information other mail systems see when trying to reach your SMTP domain.
How do I use the information? You can force NSLookup to poll a particular DNS server by using the command:
server ip_address or server fqdn
As an example, open a command prompt window
At the prompt, type nslookup
Type the command set type = mx
Type the command server a.root-servers.net
Type the registered domain name (e.g. domain.com)
You have requested the MX information for domain.com directly from one of the DNS root servers.
Here is IP information for the thirteen (13) root servers.
a.root-servers.net 198.41.0.4
b.root-servers.net 192.228.79.201
c.root-servers.net 192.33.4.12
d.root-servers.net 128.8.10.90
e.root-servers.net 192.203.230.10
f.root-servers.net 192.5.5.241
g.root-servers.net 192.112.36.4
h.root-servers.net 128.63.2.53
i.root-servers.net 192.36.148.17
j.root-servers.net 192.58.128.30
k.root-servers.net 193.0.14.129
l.root-servers.net 198.32.64.12
m.root-servers.net 202.12.27.33
The complete information can be found at the website: http://www.root-servers.org
Why would anyone care? To verify the information other mail systems see when trying to reach your SMTP domain.
How do I use the information? You can force NSLookup to poll a particular DNS server by using the command:
server ip_address or server fqdn
As an example, open a command prompt window
At the prompt, type nslookup
Type the command set type = mx
Type the command server a.root-servers.net
Type the registered domain name (e.g. domain.com)
You have requested the MX information for domain.com directly from one of the DNS root servers.
Here is IP information for the thirteen (13) root servers.
a.root-servers.net 198.41.0.4
b.root-servers.net 192.228.79.201
c.root-servers.net 192.33.4.12
d.root-servers.net 128.8.10.90
e.root-servers.net 192.203.230.10
f.root-servers.net 192.5.5.241
g.root-servers.net 192.112.36.4
h.root-servers.net 128.63.2.53
i.root-servers.net 192.36.148.17
j.root-servers.net 192.58.128.30
k.root-servers.net 193.0.14.129
l.root-servers.net 198.32.64.12
m.root-servers.net 202.12.27.33
The complete information can be found at the website: http://www.root-servers.org
Sunday, June 3, 2007
Missing messages - Part 4
The message was delivered to the mailbox - where did it go?
This is the most common scenario [I mean, speaking as one Exchange Admin to another - what else could it be? ;) ]. As a personal aside, your goal is to figure out what happened and calmly point it out to the user. The user will likely feel embarrassed already - no need to editorialize or lecture.
As discussed in Part 1, a successful message delivery typically means one of the following:
- It reached the mailbox and was segregated or deleted by a system function
- It reached the mailbox and was segregated or deleted by a client function
- It reached the mailbox and was segregated or deleted by a user function
- It reached the mailbox and was manually segregated or deleted
System function typically means forwarding configured in AD
To check for forwarding, open the Users & Computers console (ADUC) and open the properties of the recipient's object. On the Exchange General tab go to Delivery Options. Any forwarding configured at the Active Directory level will appear there.
Client functions include anti-virus/anti-spam filtering, and directing new messages to a personal folder
Check the console and logs of any 3rd-party anti-virus and anti-spam software. Check the Junk E-mail folder in the user's mailbox.
Check all workstations this user logs on to for a profile that directs all new messages to a Personal Folder instead of to the mailbox.
User functions include rules, auto-archiving, and viewing filters
Check for and disable any viewing filters in Outlook (View-->ArrangeBy-->Custom)
Check for auto-archiving (File-->Archive), look in all Personal folders listed in the Outlook profile. Search for all PST files on the local drive and all mapped drives.
Check for rules (Tools-->Rules and Alerts)
If the ruleset is empty, there is still a possibility that something formerly in rules is still acting on messages. To make sure, close Outlook, then launch it again from a command line using the /cleanrules switch (e.g. outlook.exe /cleanrules)
If the ruleset is not empty and you wish to keep them, you can export the set to a file then import again later.
Remember that the Out Of Office function can also have rules. If OOO is enabled, make sure you check that configuration for rules.
Manual processes initiated by the user
Look for and search any PST files in the Outlook profile and on the local drive.
Look in the Deleted Items folder. Look at the Recover Deleted Items area.
Search the other folders for items which were Shift-Deleted.
Missing messages - Part 3
Message Tracking sees the message, but it was not delivered to the mailbox
If Message Tracking (MT) has a record of the messsage in question, Exchange has received it. If it does not reach the mailbox, the message is typically:
- stuck in a queue
- stuck in a routing loop
- segregated by anti-virus/anti-spam filtering
You can often gain insight as to what is happening by reading through the audit trail of the message in MT.
Search the local and inter-server queues on all your servers. If found, try manually releasing it and see what happens.
Check the logs of any 3rd-party anti-virus and anti-spam software.
If Message Tracking (MT) has a record of the messsage in question, Exchange has received it. If it does not reach the mailbox, the message is typically:
- stuck in a queue
- stuck in a routing loop
- segregated by anti-virus/anti-spam filtering
You can often gain insight as to what is happening by reading through the audit trail of the message in MT.
Search the local and inter-server queues on all your servers. If found, try manually releasing it and see what happens.
Check the logs of any 3rd-party anti-virus and anti-spam software.
Missing messages - Part 2
Message Tracking does not find the message I was expecting, where could it be?
This situation calls for additional sleuthing. You need to understand your messaging environment and all the systems a message passes through on its way to the Exchange server. Identify each and check any available logs. Configuration and operation of routers, firewalls, mail gateways, even managed layer-3 switches can have an effect on inbound mail.
How widespread is this issue? Does it affect all inbound messages, a significant number of inbound messages, or a small number of inbound messages? Look for any consistencies (sending domain, sending address, receiving address).
Test inbound routing by sending yourself a message from an outside mail system (e.g. Yahoo, Hotmail, gMail). Test by sending the affected user a message from that same outside system.
This scenario can get very complicated and vary greatly from environment to environment because you are dealing with any number of different devices and configurations. Take it systematically, start at the outside and work your way in. Test each step, raise logging levels if necessary.
Missing messages - Part 1
How to start
I have been approached many times by users claiming that they never received a particular Email message. So where does one start looking? Most of the following scenarios have happened to me. The rest are follow-up thoughts of my own.
I start by asking my user some questions:
- What was the sending address?
- Approximately what time was the message sent?
- Are you seeing other messages arrive?
- If necessary may I open your mailbox to investigate?
I could also ask if the sender received a "bounce" message (a.k.a. Non-Delivery Report, a.k.a. NDR), but that tends to take extra coordinative effort. It's easier to assume that the sender is fine and to search for issues in the environment you can control (i.e. your own). Prove your own system sound before trying to look for causes outside. Show that you want to solve problems and not look for someone to blame.
Armed with this information, let's consider some possibilities:
1. It never reached our systems
2. It reached our systems but did not reach Exchange
3. It reached Exchange but was not delivered to the recipient's mailbox
4. It reached the recipient's mailbox but does not appear in the client software
The list is sorted according to message flow, but that does not mean you have to investigate in the same order.
The first question I ask myself is, does Exchange think it was delivered to the recipient's mailbox? Most of the time I find that the message did in fact reach the recipient's mailbox and something was done to it either automatically or manually.
Use Message Tracking (MT) to confirm whether the message was delivered. Use the information obtained from the user as the search parameters. If MT finds the message (regardless of outcome), rule out #1 and #2. If MT reports "Message delivered locally to store", it reached the recipient ruling out #3.
At this point, let's break the investigation into three parts.
If you cannot find the message in MT, continue with Part 2, Message Tracking does not find the message I was expecting, where could it be?
If MT finds the message, but reports something other than "delivered locally", continue with Part 3, Message Tracking sees the message, but it was not delivered to the mailbox
If MT does not find the message, continue with Part 4, The message was delivered to the mailbox - where did it go?
I have been approached many times by users claiming that they never received a particular Email message. So where does one start looking? Most of the following scenarios have happened to me. The rest are follow-up thoughts of my own.
I start by asking my user some questions:
- What was the sending address?
- Approximately what time was the message sent?
- Are you seeing other messages arrive?
- If necessary may I open your mailbox to investigate?
I could also ask if the sender received a "bounce" message (a.k.a. Non-Delivery Report, a.k.a. NDR), but that tends to take extra coordinative effort. It's easier to assume that the sender is fine and to search for issues in the environment you can control (i.e. your own). Prove your own system sound before trying to look for causes outside. Show that you want to solve problems and not look for someone to blame.
Armed with this information, let's consider some possibilities:
1. It never reached our systems
2. It reached our systems but did not reach Exchange
3. It reached Exchange but was not delivered to the recipient's mailbox
4. It reached the recipient's mailbox but does not appear in the client software
The list is sorted according to message flow, but that does not mean you have to investigate in the same order.
The first question I ask myself is, does Exchange think it was delivered to the recipient's mailbox? Most of the time I find that the message did in fact reach the recipient's mailbox and something was done to it either automatically or manually.
Use Message Tracking (MT) to confirm whether the message was delivered. Use the information obtained from the user as the search parameters. If MT finds the message (regardless of outcome), rule out #1 and #2. If MT reports "Message delivered locally to store", it reached the recipient ruling out #3.
At this point, let's break the investigation into three parts.
If you cannot find the message in MT, continue with Part 2, Message Tracking does not find the message I was expecting, where could it be?
If MT finds the message, but reports something other than "delivered locally", continue with Part 3, Message Tracking sees the message, but it was not delivered to the mailbox
If MT does not find the message, continue with Part 4, The message was delivered to the mailbox - where did it go?
Saturday, May 26, 2007
Exchange 5.5 Public Folder Tools
This isn't much of a post, but some people may find it useful. I was searching for the old PFInfo and PFAdmin tools for someone in a forum. All of the hits in my Google searching seemed to be similar queries from people trying to find the old utilities, and no reference to any online sources. The only solutions were that they were supposed to be on the Exch5.5 install CD, or that you had to call Microsoft PSS to get them.
I muddled through and finally came across a reference to our good friends at MSExchangeTeam.com and I knew I was on the right track. I can't take any credit for this, other than to say I persevered. Hopefully future search engine inquiries will find my post and this old lost tool won't be so hard to track down.
For those curious what the big deal is, the newer tool (PFDAVAdmin) doesn't work with Exchange 5.5. If you deal with Public Folder permissions, I strongly advise you check out these tools. They can save you a lot of time.
Finally, without further ado, links to all the tools can be found at: http://msexchangeteam.com/archive/2004/11/05/252979.aspx
I muddled through and finally came across a reference to our good friends at MSExchangeTeam.com and I knew I was on the right track. I can't take any credit for this, other than to say I persevered. Hopefully future search engine inquiries will find my post and this old lost tool won't be so hard to track down.
For those curious what the big deal is, the newer tool (PFDAVAdmin) doesn't work with Exchange 5.5. If you deal with Public Folder permissions, I strongly advise you check out these tools. They can save you a lot of time.
Finally, without further ado, links to all the tools can be found at: http://msexchangeteam.com/archive/2004/11/05/252979.aspx
Saturday, April 21, 2007
Recover Shift-Deleted items
I've had several Outlook users come to me asking to recover messages they've just accidentally deleted. When asked, they confess to have used Shift-Delete (permanent delete). What to do?
Outlook has the feature of being able to Recover Deleted Items, but by default it is only active for the Deleted Items folder. All other folders show the selection under the Tools menu grayed-out.
Through the registry on a workstation, you can configure Outlook to allow Recover Deleted Items for any folder.
1. Exit Outlook (if open)
2. Launch the Registry Editor (regedit or regedt32)
3. Expand HKLM\Sofware\Microsoft\Exchange\Client\Options
4. In the right pane, if there is a DWORD entry named DumpsterAlwaysOn, skip to step 7
5. Go to Edit-->New-->DWORD Value
6. Without any spaces, type the name DumpsterAlwaysOn
7. Set the DWORD value to 1 (value of 0 disables the feature)
8. Exit the registry editor
9. Launch Outlook
Outlook has the feature of being able to Recover Deleted Items, but by default it is only active for the Deleted Items folder. All other folders show the selection under the Tools menu grayed-out.
Through the registry on a workstation, you can configure Outlook to allow Recover Deleted Items for any folder.
1. Exit Outlook (if open)
2. Launch the Registry Editor (regedit or regedt32)
3. Expand HKLM\Sofware\Microsoft\Exchange\Client\Options
4. In the right pane, if there is a DWORD entry named DumpsterAlwaysOn, skip to step 7
5. Go to Edit-->New-->DWORD Value
6. Without any spaces, type the name DumpsterAlwaysOn
7. Set the DWORD value to 1 (value of 0 disables the feature)
8. Exit the registry editor
9. Launch Outlook
Gathering password change dates
This came about when my IT staff was tasked with preparing the company for mandatory password changes forced by Group Policy. We wanted to give everyone a chance to change their password once before the GPO locked them out of their accounts. For several weeks I had to gather all the password information to see who we needed to talk to.
I also wanted a distributable report, so I read up on populating Excel spreadsheets from the Microsoft Script Center website (see my links section). I included some pretty-print features like autoformatting the column widths, making the column headings bold, and performing a sort on the data.
The script has duplicate sections to pull data out of different OUs. You can add or remove as many of this section as desired. Just change the LDAP section to point to the appropriate place in your AD forest.
Copy and paste the following into a file named PASSWORDDATES.VBS then edit.
'* This script creates an Excel spreadsheet showing users and
'* their last password change date. Running this script assumes
'* you have MS-Excel loaded on the workstation.
'* - Dean T. Uemura
'* Execute the script by typing:
'* CSCRIPT PASSWORDDATES.VBS at a command line prompt
'* PasswordFile holds the name of the spreadsheet and
'* will be saved in My Documents
Const PasswordFile = "passwords.xls"
set objFSO = CreateObject("Scripting.FileSystemObject")
set objExcel = CreateObject("Excel.Application")
objExcel.Workbooks.Add
'* Establish Header Row - make them 12-point bold print
objExcel.Cells(1,1).Value = "User"
objExcel.Cells(1,2).Value = "Location"
objExcel.Cells(1,3).Value = "Password Date"
set objRange = objExcel.Range("A1:C1")
objRange.Font.Size = 12
objRange.Font.Bold = TRUE
iWriteRow = 1
on error resume next
'* Each user OU is done separately in this script
wscript.echo "OU1"
set objOU = GetObject("LDAP://ou=OU1,dc=mydomain,dc=com")
objOU.Filter = Array("user")
for each objUser in objOU
iwriteRow = iwriteRow + 1
'* wscript.echo objuser.name
dtmValue = objUser.PasswordLastChanged
objExcel.Cells(iWriteRow,1).Value = objUser.Name
objExcel.Cells(iWriteRow,2).Value = "OU1"
objExcel.Cells(iWriteRow,3).Value = dtmValue
next
wscript.echo "OU2"
set objOU = GetObject("LDAP://ou=OU2,dc=mydomain,dc=com")
objOU.Filter = Array("user")
for each objUser in objOU
iwriteRow = iwriteRow + 1
'* wscript.echo objuser.name
dtmValue = objUser.PasswordLastChanged
objExcel.Cells(iWriteRow,1).Value = objUser.Name
objExcel.Cells(iWriteRow,2).Value = "OU2"
objExcel.Cells(iWriteRow,3).Value = dtmValue
next
'* Autofit the column widths
set objRange = objExcel.Range("A1")
objRange.activate
set objRange = objExcel.ActiveCell.EntireColumn
objRange.Autofit()
set objRange = objExcel.Range("B1")
objRange.activate
set objRange = objExcel.ActiveCell.EntireColumn
objRange.Autofit()
set objRange = objExcel.Range("C1")
objRange.activate
set objRange = objExcel.ActiveCell.EntireColumn
objRange.Autofit()
'* Sort by Location then Dateset
objRange = objExcel.Range("A1").SpecialCells(11)
set objRange2 = objExcel.Range("B1")
set objRange3 = objExcel.Range("C1")
objRange.Sort objRange2,,objRange3,,,,,1
'* Save the Spreadsheet file
set objWorkbook = objExcel.ActiveWorkbook
objWorkbook.SaveAs(PasswordFile)
objExcel.Quit
I also wanted a distributable report, so I read up on populating Excel spreadsheets from the Microsoft Script Center website (see my links section). I included some pretty-print features like autoformatting the column widths, making the column headings bold, and performing a sort on the data.
The script has duplicate sections to pull data out of different OUs. You can add or remove as many of this section as desired. Just change the LDAP section to point to the appropriate place in your AD forest.
Copy and paste the following into a file named PASSWORDDATES.VBS then edit.
'* This script creates an Excel spreadsheet showing users and
'* their last password change date. Running this script assumes
'* you have MS-Excel loaded on the workstation.
'* - Dean T. Uemura
'* Execute the script by typing:
'* CSCRIPT PASSWORDDATES.VBS at a command line prompt
'* PasswordFile holds the name of the spreadsheet and
'* will be saved in My Documents
Const PasswordFile = "passwords.xls"
set objFSO = CreateObject("Scripting.FileSystemObject")
set objExcel = CreateObject("Excel.Application")
objExcel.Workbooks.Add
'* Establish Header Row - make them 12-point bold print
objExcel.Cells(1,1).Value = "User"
objExcel.Cells(1,2).Value = "Location"
objExcel.Cells(1,3).Value = "Password Date"
set objRange = objExcel.Range("A1:C1")
objRange.Font.Size = 12
objRange.Font.Bold = TRUE
iWriteRow = 1
on error resume next
'* Each user OU is done separately in this script
wscript.echo "OU1"
set objOU = GetObject("LDAP://ou=OU1,dc=mydomain,dc=com")
objOU.Filter = Array("user")
for each objUser in objOU
iwriteRow = iwriteRow + 1
'* wscript.echo objuser.name
dtmValue = objUser.PasswordLastChanged
objExcel.Cells(iWriteRow,1).Value = objUser.Name
objExcel.Cells(iWriteRow,2).Value = "OU1"
objExcel.Cells(iWriteRow,3).Value = dtmValue
next
wscript.echo "OU2"
set objOU = GetObject("LDAP://ou=OU2,dc=mydomain,dc=com")
objOU.Filter = Array("user")
for each objUser in objOU
iwriteRow = iwriteRow + 1
'* wscript.echo objuser.name
dtmValue = objUser.PasswordLastChanged
objExcel.Cells(iWriteRow,1).Value = objUser.Name
objExcel.Cells(iWriteRow,2).Value = "OU2"
objExcel.Cells(iWriteRow,3).Value = dtmValue
next
'* Autofit the column widths
set objRange = objExcel.Range("A1")
objRange.activate
set objRange = objExcel.ActiveCell.EntireColumn
objRange.Autofit()
set objRange = objExcel.Range("B1")
objRange.activate
set objRange = objExcel.ActiveCell.EntireColumn
objRange.Autofit()
set objRange = objExcel.Range("C1")
objRange.activate
set objRange = objExcel.ActiveCell.EntireColumn
objRange.Autofit()
'* Sort by Location then Dateset
objRange = objExcel.Range("A1").SpecialCells(11)
set objRange2 = objExcel.Range("B1")
set objRange3 = objExcel.Range("C1")
objRange.Sort objRange2,,objRange3,,,,,1
'* Save the Spreadsheet file
set objWorkbook = objExcel.ActiveWorkbook
objWorkbook.SaveAs(PasswordFile)
objExcel.Quit
Command Prompt window tips
Tip#1 - Entering a path+file into a command line
I find myself occasionally pining for the old MS-DOS days of typing everything from a command prompt. Then I have to type in a command with a long path and file embedded and I snap back to the present time. Here's a tip I paid for - actually I got it during a call I'd put in to Microsoft's PSS (Exchange) and this was something I was directed to do by the PSS Engineer.
1. Open a command prompt window (CPW)
2. Open Windows Explorer and drill down to a subfolder (say in My Documents)
3. Position the two windows so that when Explorer has focus, you can see at least part of the CPW
4. Click on the desired file and drag it to the CPW
Look at the CPW. You'll see the entire path+file, and if there are any embedded spaces, the entire thing is surrounded by double-quote marks! The entry gets inserted at the cursor location, so you can even place it in the middle of a typed command.
Tip#2 - Command history
If you're like me, you occasionally run into a situation where you have to repeat commands, not always in the same order. There's always the up-arrow and down-arrow to scroll through the previous commands, but there's got to be a better way right? Especially if you need to repeat a command you typed ten lines ago, then have to redo the one you just did.
Did you know that from earlier days of MS-DOS, there was a history buffer? I didn't until I was in another PSS call. With the focus on a Command Prompt window, the F7 key displays your history, and all commands are accessible by scrolling up and down with the arrow keys. Pressing Enter on any of them repeats the command.
I find myself occasionally pining for the old MS-DOS days of typing everything from a command prompt. Then I have to type in a command with a long path and file embedded and I snap back to the present time. Here's a tip I paid for - actually I got it during a call I'd put in to Microsoft's PSS (Exchange) and this was something I was directed to do by the PSS Engineer.
1. Open a command prompt window (CPW)
2. Open Windows Explorer and drill down to a subfolder (say in My Documents)
3. Position the two windows so that when Explorer has focus, you can see at least part of the CPW
4. Click on the desired file and drag it to the CPW
Look at the CPW. You'll see the entire path+file, and if there are any embedded spaces, the entire thing is surrounded by double-quote marks! The entry gets inserted at the cursor location, so you can even place it in the middle of a typed command.
Tip#2 - Command history
If you're like me, you occasionally run into a situation where you have to repeat commands, not always in the same order. There's always the up-arrow and down-arrow to scroll through the previous commands, but there's got to be a better way right? Especially if you need to repeat a command you typed ten lines ago, then have to redo the one you just did.
Did you know that from earlier days of MS-DOS, there was a history buffer? I didn't until I was in another PSS call. With the focus on a Command Prompt window, the F7 key displays your history, and all commands are accessible by scrolling up and down with the arrow keys. Pressing Enter on any of them repeats the command.
Subscribe to:
Posts (Atom)