Monday, December 29, 2014

Unhealthy Directory Synchronization Notification - Expired Credentials

As you may be aware, when there are issues with Dirsync connecting to O365, you get an Unhealthy Directory Synchronization Notification email. The email doesn't provide any information other than to check the event logs. I got this notification for my test environment recently.

I happened to have the Synchronization Service Manager (miisclient.exe) open on my Dirsync server and looked in there first. The Active Directory Connector had a status of Success, which is expected. So, all good on the local side. However, the Windows Azure Active Directory Connector had a status of "stopped-extension-dll-exception".

The Application event log gave more information:
  • Directory Synchronization, Event ID 115 - Access to Windows Azure Directory has been denied
  • Directory Synchronization, Event ID 0 - Update your password and try again
  • Directory Synchronization, Event ID 655-  Failed credential provisioning ping
The cause of my error was an expired password on the account I was using for directory synchronization. I was using a dedicated cloud account in Office 365 that I was not monitoring. The password for the account expired and was not allowing authentication for the account.

The fix was to perform the following:
  1. Log in to Office 365 as an administrator and reset the password for the directory synchronization account.
  2. Update the password in Dirsync.
To update the password in Dirsync, you can run Directory Sync Configuration again. However, instead, I updated the credentials in Synchronization Service Manager. To do this in Synchronization Service Manager, use the following steps:
  1. Open Synchronization Service Manager (miisclient.exe).
  2. In Synchronization Service Manager, click Management Agents and double-click Windows Azure Active Directory Connector.
  3. In the Properties window, click Connectivity.
  4. On the Connectivity page, update the Password and click OK.
  5. Run Start-OnlineCoexistenceSync (or wait for Dirsync to do it automatically)

Update Dirsync Credentials

The long term fix for this issue to prevent the Dirsync credentials from expiring on the account. In the graphical interface of O365 management, you can configure a password expiration policy for all cloud accounts but not individual cloud accounts. To set the password expiration policy for a single account, you need to use Windows PowerShell and the command Set-MsolUser -UserPrincipalName -PasswordNeverExpires $true.

A complete set of instructions is here:

Friday, December 19, 2014

Updated Exchange Online Protection Addresses - Jan 2015

Microsoft has indicated that they are going to be updating the Exchange Online Protection addresses in January 2015. Here is a link with the new IP addresses:
What does this mean to you? Well, these are the outbound addresses from EOP to your on premises email system. So, if you are using EOP for your on-premises email system, you need to ensure that your firewall is updated to allow traffic from these IP addresses to your on-premises email system.

Also, it is not explicitly stated, but I would be concerned that mailflow for hybrid environments will be affected. So, it may be necessary to go into the receive connector used for hybrid mailflow and allow these IP addresses there also. I'll try to check on my hybrid system when the changes go into effect in January and update this post.

Saturday, December 6, 2014

Terminal Services Device Redirector Missing

We have a client using SBS 2008 that is unable to use printer redirection to print locally when connected to the server via RDP. Printer redirection is basic functionality in RDP and normally works without any setup at all. We took over maintenance on this server from another organization about 4 years ago, but never had any need to test or use printer redirection until recently.

After taking a look at the event logs, we saw EventID 1103 for Microsoft-Windows-TerminalServices-Printers with the following message:
An internal communication error occurred.  Redirected printing will no longer function for a single user session.  Check the status of the Terminal Services Device Redirector in the System folder of Device Manager.
The Terminal Services Device Redirector is a device driver that is responsible for printer redirection. On this server, the Terminal Services Device Redirector did not exist in Device Manager. So, we need to install it.

Conveniently, several web sites provide instructions on how to use the devcon utility to install the RDPDR driver and get this going again. However, many of these are for Windows Server 2003. On Windows Server 2008, which is 64-bit, the included devcon utility doesn't work properly because it is 32-bit. The 32-bit version of devcon can query drivers, but not change or add them. It will give a delightfully generic error message of "Devcon Failed".

So, step 1 is to download the 64-bit version of devcon. It is included as part of the Windows Driver Kit. This article describes how to download the 600 MB Windows Driver Kit and extract only the 64-bit version devcon from it:
After you have the correct version of devcon, installation of the Terminal Services Device Redirector is easy. Run the following command and then restart the server (note the space between inf and root):
  • devcon -r install %windir%\inf\machine\inf root\rdpdr
After a reboot I was able to redirect printers properly. I have no idea how this went missing in the first place.

Microsoft has a TechNet article about this event ( but I found at least one item in it may be inaccurate. The article states that there should be a registry key for the RDPDR at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\RDPDR. The system I was working on did not have this registry key before or after I fixed it by installing the Terminal Services Device Redirector. Instead, there was some information added to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\Root\System\0003. This may be because I added it manually.

The article also mentions the idea of reinstalling Terminal Services as a fix. Given that this server did not have Terminal Services installed, I'm not sure how we would have accomplished that. It had remote access enabled in Admin mode, but not full on Terminal Services.

If what I have written doesn't fix your issue with Terminal Services printing, take a look at this article by It has a whole bunch of issues to look at:

Friday, November 28, 2014

Outlook 2013 - Constant restarts required

During a recent migration from Exchange 2007 to Exchange 2013, we ran into an issue that I was unaware of. Outlook 2013 was popping up multiple times per day with the following message:
The Microsoft Exchange administrator has mad a change that requires you to quit and restart Outlook.
After doing some research this appears when the default public folder database for a Mailbox database is set to a non-existent public folder database. In our scenario, we removed Exchange 2007 public folders after introducing Exchange 2013. Consequently, the Exchange 2013 databases were configured to use the Exchange 2007 public folder database as default. After removing the Exchange 2007 public folder database, the Exchange 2013 database were still configured to use the deleted public folder database that was now in deleted objects.

This issue seems to be specific to Outlook 2013 and Exchange 2013 CU5. I have not applied CU6 because there are known compatibility issues with Exchange 2007 coexistence for ActiveSync.

The fix for this was to use ADSI Edit and clear the msExchHomePublicMDB attribute on the database object. The databases for Exchange 2013 are located in: CN=Configure,DC=domain,DC=com > CN=Services > CN=Microsoft Exchange > CN=OrganizationName > CN=Administrative Groups > CN=Exchange Administrative Group (FYDIBOHF23SPDLT) > CN=Databases.

Properties of Mailbox Database in ADSI Edit

For more information about this issue, see:
UPDATE: We were also running into a strange issue with Outlook 2007 where clients would connect and be able to send messages immediately but not send for a few minutes. That issue was also resolved by removing the reference to the old public folder database.

Monday, November 24, 2014

Searching the Message Tracking Log for Virus Recipients

We had a client get infected with a virus today that used Outlook for sending messages out. This was unusual. Most viruses attempt to deliver email themselves and can be blocked by the firewall. Because this virus used Outlook, the messages were sent through the Exchange server which is allowed to deliver email to the Internet.

Three specific users got infected and I wanted to be able to inform those that were sent messages not to open them. I could get this information from the message tracking log based on the subject of the message. I used three commands to dump the information to a text file.

First, get the list of messages sent by a specific user:
$UserMessages=Get-MessageTrackingLog -Start "MM/DD/YYY 00:00:00AM" -Resultsize Unlimited | Where-Object {$_.MessageSubject -like "SubjectOfVirusMessage" -and $_.EventID -like "Send" -and $_.Sender -like "SenderEmailAddress"

Second, build a list of message recpients:
ForEach ($m in $UserMessages) {$UserRecipients=$UserRecipients+$_.Recipients}

Finally, dump to text file.
$UserRecipients > C:\UserRecipients.txt

I then provided the text file to each user to inform the necessary recipients.

Removing a Message from All Mailboxes

This morning a client got a virus infected email message that got sent to all users. Only 3 users opened the infected attachment initially, but I didn't want to count on users to delete the message manually. The risk of one forgetting and opening the attachment was too high.

In this case the virus was in an attachment named This filename was unique enough that I was comfortable searching for all instances of that attachment and deleting the messages. If the filename was more generic, I would have included the message subject in the query.

Before you can perform this type of search, your user account must be a member of the Discovery Management group. This is required to do multi-mailbox search. Group membership does not take effect immediately, you may need to close and reopen your Exchange Management Shell prompt.

The syntax I used to delete all messages with the specific attachment from all mailboxes was:
Get-Mailbox -Resultsize Unlimited | Search-Mailbox -SearchQuery "" -DeleteContent
Note: This deletes the entire message, not just the attachment.

Friday, November 21, 2014

Missing Email Attachment

A few weeks ago, a client reported that when sending to a list of email addresses, one of the recipients was not getting the attachment. The recipient got the email message, but not the attachment. So, then you start to work it through logically:
  • Others get the attachment, so, it must not be on our end.
  • Send a test message from a non-client account and the attachment goes through, so, it must be on our end.
  • Apparently it's nobody's fault.
Of course it's always someone's fault, and it this case, I'm blaming Outlook. This turned out to be a new problem I've never seen with rich text formatting (RTF).

I'm sure that everyone that supports Outlook has at one time or another run into the winmail.dat attachment problem when Outlook sends RTF formatted email to a client that doesn't understand it. In this case, the client receiving the email partially understood RTF. It understood it enough to get the text message out, but not the attachment. The whole message was being delivered, but the client software dropped the attachment.

Outlook allows you to define on a per contact basis what formatting will be used when sending a message to a recipient. I'm yet to find a user that sets this on purpose, but in this case, it was accidentally set for RTF.

Rather than trying to fix this at the Outlook level and to ensure that it never happens again, I've disable RTF at the Exchange server level. If the client sends an RTF formatted message, Exchange will change it to HTML which is much more widely supported.

In Exchange Server, you can set how RTF format is used per domain. By default, there is a single domain in Organization Configuration > Hub Transport > Remote Domains, named Default. On the Message Format tab for this domain, under Exchange rich-text format, select Never Use. After doing this attachments were delivered correctly.

Properties of Default Remote Domain

Monday, November 17, 2014

Disable Inbound Proxy Probe Messages for Journaling

During a recent Exchange 2007 to Exchange 2013 migration, we ran into an issue with Managed Availability in Exchange 2013. Managed Availability provides health monitoring services.

This particular client is performing journaling by enabling journaling on each database and configuring all messages to be sent to a journaling mailbox in a separate database. They then have archiving software that removes the messages from the journaling mailbox. This allows them to retain a copy of all messages sent or received in the organization.

As we implemented Exchange 2013, the journaling mailbox was filled with messages generated by Managed Availability. The messages were from with a subject of Inbound proxy probe to a health mailbox.

Managed Availability was sending messages through the transport system to health mailboxes to verify that the system was functioning normally. However, this was resulting in a lot of messages that were not required being journaled. To avoid this, there are two options:
  1. Use a Journaling rule for specific users or group membership instead.
  2. Disable some monitors in Managed Availability.
I'm always reluctant to disable basic functionality in the system. So, our first attempt was by using a journaling rule with a dynamic distribution group for all users. When we did this we found that it was not reliably journaling the messages. So, we abandoned the journaling rule option and chose to disable the monitors generating the email messages instead.

In a knowledgebase article, Microsoft provides the following three commands for disabling the messages:
Add-GlobalMonitoringOverride -Identity "FrontendTransport\OnPremisesSmtpClientSubmission" -PropertyName Enabled -PropertyValue 0 -ApplyVersion "15.0.620.29" -ItemType Probe
Add-GlobalMonitoringOverride -Identity "MailboxTransport\Mapi.Submit.Probe" -PropertyName Enabled -PropertyValue 0 -ApplyVersion "15.0.620.29" -ItemType Probe
Add-GlobalMonitoringOverride -Identity "FrontendTransport\OnPremisesInboundProxy" -PropertyName Enabled -PropertyValue 0 -ApplyVersion "15.0.620.29" -ItemType Probe
In these commands, you must provide the -ApplyVersion parameter which specifies which version of Exchange Server that the override applies to. When you apply new updates to your system, you must run these commands again.

To identify the version of Exchange you are running, you can use the following command:
Get-ExchangeServer | FL Name,AdminDisplayVersion
The Microsoft documentation does not talk about it, but unless you have a single database, you have more than one OnPremisesInboundProxy monitor. There is an OnPremisesInboundProxy monitor for each database and is unique for the health mailbox associated with each database and you need create an override for each one. On this system with four mailbox databases, the identity of each monitor was:
  • FrontendTransport\OnPremisesInboundProxy
  • FrontendTransport\OnPremisesInboundProxy_2
  • FrontendTransport\OnPremisesInboundProxy_3
  • FrontendTransport\OnPremisesInboundProxy_4
To identify all of the OnPremisesInboundProxy monitors on your system, you can use the following command:
(Get-WinEvent -LogName Microsoft-Exchange-ActiveMonitoring/ProbeDefinition | % {[XML]$_.toXml()}).event.userData.eventXml | ?{$_.Name -like "OnPremisesInboundProxy*"}
After creating an override for each instance of OnPremisesInboundProxy, all of the Inbound Proxy Probe messages were disabled.

To ensure that the monitoring overrides take effect, you need to restart the Microsoft Exchange Diagnostics and Microsoft Exchange Health Manager services on all Exchange 2013 servers.

Additional resources:

iDRAC7 - No Mouse or Keyboard

We recently had an emergency issue with a client system and had issues with the iDRAC7 remote control features. We put iDRAC cards in all of our servers for exactly this scenario where there is an unknown issue and we might need to power cycle the system.

We were able to log into the iDRAC7 web console with no issues. However, on the summary screen it was not showing status for the power supplies. Also, if I tried to browse the disk status, it was indicating the no disks or controllers could be seen.

Fortunately, the fix was pretty easy. Just restart the iDRAC card. To do this, on the Summary screen, in the Quick Launch Tasks, click Reset iDRAC as shown below. This does not reset the configuration of the iDRAC, it just restarts the card.

Wednesday, November 5, 2014

Require Encryption for a Specific Email Domain

You might not realize it, but your Exchange server is probably already encrypting messages sent and received from the Internet. By default, Exchange uses opportunistic TLS. This means that it offers TLS for inbound messages, but does not require it. Exchange also tries to use TLS for outbound messages, but does not require it.

We have one client that works with an insurance company. In order to ensure that data is secure, they request that their customers force the use TLS instead of relying on opportunistic TLS. This is more secure because the messages will wait in the queue if TLS cannot be established.

To use TLS for inbound messages, you need to have a valid certificate installed on your Exchange server and have assigned the SMTP service to that certificate. That certificate needs to include the name that external servers use to reach your server, such as As long as Transport Layer Security is enabled as an authentication mechanism on the Receive connector, opportunistic TLS is used for inbound messages.

You do not need to do anything to use TLS for outbound messages. TLS for outbound message relies on the certificate of the recipient server. However, you can enforce the use of TLS for specific domains by creating a send connector for those domains. Then after the send connector is created, you can use the Exchange Management Shell (EMS) to for TLS for that send connector by using the following command:
Set-SendConnector TLSConnector -RequireTLS $True
You can also force TLS for a receive connector, however, those are based on IP address. If the sender changes the IP address, then TLS will not longer be required. So, in most cases opportunistic TLS is a better choice for inbound messages.

Note: If you have another proxying device like an antispam appliance between Exchange and the Internet then you need to setup encryption on that device rather than your Exchange server. 

Monday, October 27, 2014

Exclaimer Attaching Logo Instead of Showing Inline

We have a client that wants a very specific email signature attached all outbound email messages. They are a fairly small client with about 20 users. So, manually configuring Outlook worked for them. Where it fell apart was all the mobile devices that they use. They have both iPhones and iPads for various users.

By default iPhones and iPads don't sent HTML email. And there is no built-in method for creating messages with graphics such a logos in the signature. I did have some success copying an HTML signature from an existing email and pasting it into the signature for an iPhone, but the results were inconsistent. Some devices properly sent the signature with the logo and other didn't. Even more frustrating, sometimes it started working fine and then stopped.

The solution is to implement Exclaimer Signature Manager for Exchange ( on the Exchange server. Exclaimer manages the signature at the server level instead of at the device level. So, we don't need to worry about the type of device. It can also convert existing messages to HTML to allow the graphic logo to be embedded.

The Exclaimer software is very simple to install (almost too easy). The editing of signatures is quite intuitive and pretty easy to customize. However, I ran into problems when I was testing it.

I created a template that included the logo and applied it messages only for the Administrator account. When I sent messages, the logo appears as an attachment rather than inline in the text as it should. It looked like this:

When I was searching around, the only references I could find to logos not appearing properly suggested that it was because there were incorrect permissions for the the logo file. Network Service needs Read access to the logo. This didn't seem to apply here because the logo was being read, it just wasn't embedded properly. However, even when I gave Everyone Full permissions to the logo file, there was not fix. I also tried restarting services and disabling the antispam software running on that server. All with no fix.

If you look at the source html in the message, it does refer to the logo file with the following code, but can't seem to find it:
Fortunately, Exclaimer support was pretty quick to help me out. This server has a disclaimer configured in the hub transport rules. After disabling the disclaimer in the hub transport rules the logo was properly embedded. So, there must be an issue when both Exchange and Exclaimer try to manipulate the HTML content in the message.

Wednesday, October 22, 2014

Kill and Restart Hung Services by Using PowerShell

We have a client with a misbehaving service. An older version of Tomcat that runs a web application. The Tomcat service tries to restart itself at 1am each night but as often than not, it ends up hanging in the stopping phase. At this point, I manually kill the process and start the service.

After going through this process for a while, I figured we better script this and make it a scheduled task. Here is the script:
$tom = Get-Process tomcat6
Stop-Process $
Start-Sleep -s 5
Start-Service tomcat6
The script loads the process tomcat6 into the variable $tom. Then it stops the process based on the process id of $tom. If you try to pass the entire process object it fails.

Then the script pauses for 5 seconds and starts the tomcat6 service. I found that if I didn't have a pause between killing the process and starting the service, the service wouldn't start. I'm assuming this is because the process is not stopped entirely before it gets to starting the service. Putting in the pause ensures that the process is complete stopped before starting the service.

When setting up a PowerShell script as a scheduled task, you use the following options:
  • Start a program
  • Program/script: powershell
  • Add arguments: .\nameofscript.ps1
  • Start in: C:\scriptfolder

Tuesday, October 21, 2014

Extension Custom Attributes for Dynamic Distribution Lists in Office 365

I was browsing an online forum recently and saw someone with an issue creating dynamic distribution groups in Office 365. The mailing lists were based on the Office property. However, to accommodate users that work out of multiple locations the poster was trying to put a comma separated list in the Office property. Then for the distribution group the poster wanted to do a recipient filter including something like:
Office -like '*location1*'
This syntax works properly for on-premises Exchange but won't on Office 365. I assume that it is because Office 365 is trying to limit the load these queries place on the infrastructure.

In Office 365, you can't start your query with a wildcard. So, you can search for 'location*' but this only returns true if the value is at the start of the string.

The second bright idea I had was to fake it out and put a dummy value first. The I could use two wildcards to get the same effect. So, the query would be for 'X*location1*' but this didn't work either. There was no error for this one, but it never returned true. I suspect that the query is just tossed in the background.

Then I found out about some new attributes that were added starting with Exchange 2010 SP2 and are also in Exchange 2013/Office 365:
  • ExtensionCustomAttribute1
  • ExtensionCustomAttribute2
  • ExtensionCustomAttribute3
  • ExtensionCustomAttribute4
  • ExtensionCustomAttribute5
These 5 attributes are not the same as the 15 CustomAttributes that have in Exchange Server for many versions. The CustomAttributeX properties are a single text string. The ExtensionCustomAttributeX properties are an array. This means that they can store multiple values that can be properly queried for a dynamic distribution group.

You can set multiple values for an extension custom attribute as follows:
Set-Mailbox Byron -ExtensionCustomAttribute1 London,Paris,Montreal

If you need to add or remove individual items to the attribute for a user, you can use the following syntax:
Set-Mailbox Byron -ExtensionCustomAttribute1 @{Add="Vancouver"; Remove "London"}

Creating the dynamic distribution list for recipients with Vancouver as an item in ExtensionCustomAttribute1 would be as follows:
New-DynamicDistributionGroup VancouverList -RecipientFilter {ExtensionCustomAttribute1 -eq "Vancouver"}
A final quirk about ExtensionCustomAttributeX and Dirsync. When you use Dirsync (Windows Azure Active Directory Synchronization), it will appear that ExtensionCustomAttributeX is syncing up to Office 365. However, it will never appear in the properties of the users. Somewhere in the background Windows Azure AD or Exchange Online discards the value. This is a known issue.

Monitor the following forum posts to see if there is any change:

Sunday, October 12, 2014

Change UPN After Rename with Dirsync

It's pretty common to rename user accounts in your organization. This is typically done when a person gets married. In a large organization, this happens fairly often. If you are syncing with Office 365, you want to change the user id there also. Unfortunately, this is a manual process. Let me explain....

Let's say that I have a user with the UPN of in my organization. When I first run Dirsync, this UPN is synchronized in Azure AD and Office 365. I can not use this UPN for authenticating to both on-premises services and Office 365.

Several months later I meet the lady of my dreams and get married. Now I change the on-premises information including the UPN. So, my new UPN on-premises is The updated last name, display name, and UPN show as synchronized when I view the logs in DirSync. However, in Office 365, the new UPN never appears. In Office 365, I still need to authenticate as the original UPN of

While this behaviour is surprising the first time you run into it, it is normal. The domain for your UPN can be updated by using DirSync, but not the username portion. Instead, to update the UPN, you need to do a remote connection to Office 365 and run the following command:

Set-MsolUserPrincipalName –UserPrincipalName oldUPN –NewUserPrincipalName newUPN
In my example the command would be:
Set-MsolUserPrincipalName –UserPrincipalName –NewUserPrincipalName
UPDATE: This information is no longer accurate. I'm not sure when it changed, but during a migration yesterday for a client and a test I performed today (March 21/15) the username is now properly updated by DirSync. You should not need to run Set-MsolUserPrincipalName when using Dirsync.

UPDATE: And on a project today (June 23/15) we had issues with UPN sync and needed to use Set-MsolUserPrincipalName. Not sure if MS changed something or whether I was out of my mind previously.

Friday, October 10, 2014

Exclude Resource Mailboxes from Address Lists and Dynamic Distribution Groups

In Exchange 2010 when you generate an address list or dynamic distribution group containing Users with Exchange mailboxes this includes room and equipment mailboxes. If you want to exclude resource mailboxes, you'll need to use PowerShell to set the recipient filter.

Here is a command that creates an address list for users with Exchange mailboxes, which includes resource mailboxes:
New-AddressList "MailboxUsers" -RecipientFilter {(RecipientType -eq 'UserMailbox')}

Here is a command that creates an address list for user with Exchange mailboxes, but excludes resource mailboxes:
New-AddressList "MailboxUsersNoRooms" -RecipientFilter {(RecipientType –eq ‘UserMailbox’) –and (-not(ResourceMetaData –like ‘ResourceType:*’))}

There is an oPath filterable property named IsResource however, I couldn't use that for address lists because there is no LDAP equivalent and the cmdlet errored out.

Thursday, October 9, 2014

Find Stale Computer Accounts in Active Directory

The simplest way to find old unused computer accounts is by using a PowerShell query. You can use Get-ADComputer to do the query. In smaller environments, you can do a simple query for all computer accounts sorted by LastLogonDate. This query puts the oldest logon dates at the top:
Get-ADComputer -Filter * -Properties LastLogonDate | Sort-Object LastLogonDate | Format-Table Name,LastLogonDate
The -Filter parameter is required, by using an asterisk, you are querying for all computer accounts. You need to use the -Properties parameter because the Get-ADComputer cmdlet doesn't query for all computer account properties by default. So, you can use the -Properties parameter to specify that LastLogonDate should be retrieved.

Be aware that servers will be included in this list and that LastLogonDate is not entirely accurate when identifying whether servers are in use. For example, I just did a query for a client with an active application server that shows the LastLogonDate as being three months ago. However, I know for sure that clients are actively using the application on that server.

The idle time for computers in your organization may vary. So, for desktop computers 3 months or so is probably a good guideline for identifying unused computer accounts.

In a larger environment you don't want to see all of the computer accounts listed by your query. Instead you want to see only the accounts that you may be concerned about that haven't logged on for a certain timeframe. The command below queries only computer accounts that have not logged on for 90 days.

Get-ADComputer -Filter * -Properties LastLogonDate | Where-Object {$_.LastLogonDate -lt (Get-Date).AddDays(-90)} | Sort-Object LastLogonDate | Format-Table Name,LastLogonDate

Wednesday, October 8, 2014

Port 25 Blocked for Specific Domains

This was such a strange issue that I don't know if this post will ever help anyone. I just need to write it for therapy......

Starting on Friday of last week, I got reports from one client that some emails were not being delivered. This has happened to clients in the past when they were on block lists. So, my first step was to check some block list tools to see what's up. Each of the block list tools indicated that the IP was not being blocked.

Some of the antispam appliances have block lists that are not checked by the typical web sites. So, I tried to dig into it a bit further. One of the sites that was being blocked is a local university and I know several of the server/email admins there. So, emailed, explaining that I figured it was an antispam issue. They sent me the site to check block list for their appliance and it came up clean. In fact, they said my request for delivery wasn't showing up in the logs at all.

I should also note that access to ports other than 25 was unaffected. You could do ping the IP of the mail servers and connect to websites on that IP if a website was there. Only port 25 connectivity was blocked.

One of the things I had tried to do was use Putty (a free telnet client) to telnet to some of the failing email servers. Often the email server provides a message that indicates which antispam product is being used so that you can investigate why you're being blocked. In this case, each of the attempts to connect ended with a dropped connection.

Of course my other concern is that the firewall has gone nuts and is blocking things that it shouldn't. So, I did some logging and testing of rules. Nope, didn't look like the firewall was blocking anything.

As the suggestion of the admin at the university, I called the ISP. The admin at the university indicated that they had a call from someone else on Bell that couldn't deliver to them also. Weird....

So, next step is call the ISP. In the past my experience with ISPs and tech support has been rather poor overall. In this case Bell was great, they took the info and began investigating. They didn't blame my configuration. They also came back and indicated that it was their connectivity through Shaw that was the issue. So, anything being delivered to Shaw or routed through the Shaw network was affected.

It took about 5 days for Bell to convince Shaw that the issue was on their network, but the problem is now resolved. My best guess is that Shaw was marking the traffic from Bell like they do consumer Internet tracking and blocking port 25.

Need to Find Your Android Device?

This web page allows you log on with your Google account (which was required when getting the device online) and shows the location of your device. You can wipe it if you lost it. You can also ring it to find it in your house.
The hard part is remembering the credentials for your Google account if you don't use it very often.

Monday, October 6, 2014

Script to Resolve Error When Running Enable-RemoteMailbox

If you have existing user accounts in your hybrid environment, and want create a mailbox in Office 365 for those users, you can use the Enable-RemoteMailbox cmdlet. However, when you try to use Enable-RemoteMailbox you commonly get the following error:

The address '' is invalid: "" isn't a valid SMTP address. The domain name can't contain spaces and it has to have a prefix and a suffix, such as
This error occurs because the cmdlet is not building the RemoteRoutingAddress properly. This address is used for routing messages to the Office 365. So, we need to create that RemoteRoutingAddress property as part of a piped command or script.

I've seen several examples using piped commands, but I prefer a script because I find it easier to follow the logic. Here is the script I used recently:
$users = get-user -OrganizationalUnit "OU=xx,DC=domain,DC=com" -RecipientTypeDetails User 
foreach ($u in $users) {
   $alias = $u.FirstName + "." + $u.LastName
   $RR= $alias + ""
   Enable-RemoteMailbox -identity $u -RemoteRoutingAddress $RR
This script obtains a list of users without mailboxes from a specific OU. Then it loops through that list of users, takes user properties to build the remote routing address in the variable $RR. Finally it enables the remote mailbox for each user.

When you use the Enable-RemoteMailbox cmdlet, it also automatically adds that remote routing address as an email address for the account. Which is of course required for Office 365 to accept the mail.

Friday, October 3, 2014

Exchange Hybrid Mode and Dynamic Distribution Groups

Exchange Online/Office 365 does not have dynamic distribution groups. So, in a hybrid deployment, it's not possible to synchronize dynamic distribution groups from on-premises to Office 365. There are two work arounds:

Option 1
If you like scripting, you can create a script that updates the membership of a normal distribution group. You'll need to run the script as a scheduled task. The main benefit to this method is that it is contained entirely within the on-premises environment.

There are two drawbacks to this method:
  1. It's not actually dynamic, so there is a lag time from when new members are created and when they're added to the group.
  2. It's relatively complex to create the script and schedule a powershell script to run as a task with the correct snap-ins loaded.
Option 2
My preferred option for this is to create a contact in Office 365 that points at the dynamic distribution group on-premises. This allows you to continue using true dynamic distribution groups on your on-premises environment and give Office 365 users the ability to send messages to them.

When you create the dynamic distribution group you need to select the following recipient types:
  • Users with Exchange Mailboxes. These are your on premises users.
  • Users with external e-mail addresses. These are your office 365 users.
After you create the dynamic distribution group, collect the following information from it:
  • Display Name
  • Alias
  • Email address (not the address)
Now use the Office 365 admin console to create a mail contact (do not create on-premises contact and sync it) with the following attributes:
  • Display Name: same as dynamic distribution group
  • Alias: same as dynamic distribution group
  • External email address: same as email address from dynamic distribution group
From a management perspective, this is a bit of a pain because you need to remember to manually create an extra contact in Office 365. However, I still think this is the easiest way to get it going.

You can't create the contact locally and sync it for two reasons:
  1. You can duplicate the same display name on two objects. So, you'll end up with two objects using the same display name. This will be confusing for on-premises users who will see both objects.
  2. You can't duplicate the external email address for a contact with the actual email address for the dynamic distribution group. This makes it not just a bad idea, but a technical impossibility.

Exchange Online Limits

When you implement Office 365, you are using Exchange Online for email. Exchange Online is regularly changing the limits that apply for things like the size of the mailbox and attachment sizes. This web page is maintained with the up to date information on limits for Exchange Online:

Sunday, September 28, 2014

Trigger a Full Sync of Passwords for DirSync

Normally the only time you need to do a full synchronization of passwords with DirSync is after you install it. By default, when you complete the configuration wizard it performs a full synchronization of passwords. After that point passwords are synchronized only accounts are created or when the password is changed.

One potential issue that can pop up is changed passwords on the Office 365 side. When passwords are synchronized to Office 365, it is still possible to change them in Office 365 for the user account. This is not recommended from a management perspective, but administrators make bad choices once in a while. When this happens the account is out of sync between on-premises and Office 365.

To ensure that all passwords in Office 365 match their on-premises account, you can trigger the same type of full password synchronization that occurs after DirSync is installed. Perform the following steps:
  1. Open a Windows PowerShell prompt.
  2. Type Import-Module DirSync and press Enter.
  3. Type Set-FullPasswordSync and press Enter.
  4. Use the Services console to restart the Federation Identity Manager Synchronization Service.

Note: If you try to use the Restart-Service cmdlet to restart the Federation Identify Manager Synchronization Services, you will need to use the -Force parameter and it will not restart the Windows Azure Active Directory Sync Service properly.

You can verify the synchronization was successful by looking for Event ID 657 in the Application event log that shows the passwords being synchronized.

Wednesday, September 24, 2014

Firewalls and Proxy for Exchange 2010 Hybrid Mode

Most of the time when we configure hybrid mode, the CAS servers have unrestricted access to the Internet. However, not all organizations allow this. I recently ran into a couple of proxy configuration issues when setting up hybrid mode for a client.

The first errors we got were during the setup of hybrid mode creating the federation trust. However, we would have had ongoing issues because the CAS servers in the on-premises Exchange installation need to communicate with O365 servers to perform tasks such as free busy lookups. I had been assuming the CAS servers had direct Internet access but forgot to confirm with the client.

When creating the federation trust initially we ran into the following:
Unable to access the Federation Metadata document from the federation partner. Detailed information: "Unable to connect to the remote server."
The good news is that error instantly makes you think network connectivity. This error is generated when the the CAS server cannot communicate with the following URL:
In this case the client uses a proxy server for all outbound http/https traffic. Exchange server will not use the proxy configured in Internet Explorer/Windows. Instead you need to manually configure the proxy server used by Exchange server with the following cmdlet:
Set-ExchangeServer -Identity CASServer -InternetWebProxy:http://proxy:port
After setting the proxy for Exchange Server, we got an error:
(407) Proxy Authentication Required
At this point, there really is nothing we can do because there is no way for Exchange Server to provide authentication to the proxy server. So, we worked with the network team to modify the proxy configuration to not require authentication for the CAS servers. They configured those exceptions based on the IP address of the CAS servers.

For this particular organization, the other alternative would be allowing the CAS servers to communicate directly out, but only to the specific servers required. Microsoft provides a list of Exchange Online and Federation Proxy server IP Addresses to restrict outbound communication if required, but this is a large list and there is the possibility of change over time. This blog post has the links to the IP addresses:
Based on my research, another issue that can occur related to proxy settings is the use of WPAD (Web Proxy Autodiscovery Protocol). Apparently Exchange Server will use this if it's available, even if you have manually set the the proxy with Set-ExchangeServer. So, you may need to disable that if it's in use. Fortunately, this is an easy one, just stop and disable the WinHTTP Web Proxy Auto-Discovery Service.

Our second issue with the proxy server occured when we were adding Exchange Online as a forest in the Exchange Management console.I neglected to get the exact error message, but it was similar to this:
The attempt to connect to using "Basic" authentication failed. Connecting to the remote server failed with the following error message: The WinRM client is unable to connect.
The good part about this error message is that it tells us what software in unable to connect. Seeing that it's the WinRM client gave us an indication that this is likely the same type of proxy issue that we had with the Federation Trust. However, since the WinRM client is not part of Exchange Server the Set-ExchangeServer cmdlet we used earlier had no effect. Instead we used netsh to import the proxy settings from IE for use by all windows services.
netsh winhttp> import proxy source = ie
I'm not sure whether the WinRM client runs with the security credentials of the user or not. We ran this command on the CAS servers so there were no issues with authentication to the proxy because the proxy already allowed the CAS servers directly out without authentication.