Sunday, August 13, 2017

Scripting Complexity vs. Simplicity

Note: Most of my blog posts are technical items that relate to performing a specific task or fixing a specific error. This one is more of an opinion piece. So, if my arguments sound reasonable, take the advice. If you disagree, by all means, go your own way.

I am a big believer that I need to understand the details of any script I run in a production environment. Unless that script is from Microsoft and provided to perform a specific task (such as migrating public folders), I will go through a script line by line to verify I understand it. This even applies to my own scripts. If I haven’t used a script for several months, I’ll review it before I use it again to make sure I know exactly what it’s doing.

I expect that most system administrators operate with the same requirement to understand the scripts they are running. At least I hope they do. I don’t want anyone blindly running a script I created without understanding the script and knowing what it will do in their environment.

So, from my perspective, running a script is very different from running a program/utility. A program is distributed as an executable and you simply need to trust the developer. I don’t need to trust a script writer if I can understand what the script is doing. The writer just saved me the effort of creating the script myself. It’s much faster to review and verify an existing script than it is to create one.

Simple is good, right?

If you need to understand a script, then simple is good. However, we often attempt want a script to be resilient to user error. That is, we want the script to do things like:
  • Not accept invalid values
  • Not make incorrect/invalid changes to objects
  • Warn when mistakes are about to be made
A more user-friendly script is good, but it becomes more complex and more difficult to interpret. So, does that mean complexity is good?

Know your audience

The balance of simple (but you need to know what your doing) and complex (with lots of error checking) depends on your audience. If the audience needs to understand the details of your script, then simplicity is better. If the audience does not care about the details inside the script then simplicity is less important. For example, if the help desk staff in your organization run standardized scripts to complete some specific tasks then user friendly is more important than simplicity. The help desk staff will be running the script without reviewing the contents or understanding how it works. So, if they give an invalid value, you need to account for that in your script. It’s worthwhile to make it user friendly to prevent errors in the production environment.

The customers I work with typically will look at my scripts before running them. So, these scripts need to be relatively short and understandable. The documentation that I provide for running these scripts also needs to be short and understandable. In this case simplicity is most important for the audience. For example, if there are several tasks that need to be accomplished such as updating email addresses, UPNs, etc. I will create separate scripts for each task. This makes each individual script easier to understand and document.

Striking a balance

I tend to work with customers that have varying knowledge levels and needs. Some of them would run a script without looking and others want a detailed understanding how the scripts work. I want my scripts to work in both scenarios. So, I need to put in some of the user-friendly bits like prompting for values when required and some basic checking for invalid values, but I don’t try to account for every possible error. To account for every possible error would be too much complexity.

Because I can’t (or won’t) put in tons of error checking, one thing I often do is display a confirmation on screen before performing an action. For example, I have a script to remove email addresses for a specified domain. Before script performs the action, it displays on screen the pattern being searched for and provides an email address from the first mailbox as an example of what is about to be removed. The script also displays the number of mailboxes that are being modified. Displaying this information allows the user to provide a final sanity check before approving removal from all mailboxes.

To make scripts more understandable, I use lots of comments that describe what each section of a script is doing.  This is useful for customers that want to review the script. It’s also useful for me when I want to review the script contents or modify the script. Embedding comment-based help in a script makes it easier to use, but detailed comments in the script are better for understanding how it works.

If you don’t get the right balance then your scripts won’t get used. That help desk person will think your script generates errors all the time and ignore it. The administrators you gave complex scripts to will build their own because they can’t understand yours.

Friday, June 30, 2017

June 2017 Security Update Breaks Outlook Search

The June 2017 security update for Windows causes problems for search in Outlook clients using cached mode.  Uses will see incomplete search results and may be notified that:
Search results may be incomplete because items are still being indexed.
This issue affects Windows 7, Windows 8.1, and Windows 10. It also affects all versions of Outlook.


Microsoft has fixed this issue with a new update released on June 27th. You can download the update here:
These fixes are for retail, OEM, and volume licensed versions of Outlook. If your Outlook is included with an Office 365 version of Microsoft Office, then updates are already available and should be installed automatically. If they are not installed automatically performed the following steps:
  1.  In Outlook, click the File tab, and click Office Account.
  2. Click Update Options and click Update Now.

To review Microsoft's documentation on this issue, see Issue #5 on this page:

Wednesday, June 28, 2017

Exchange 2010 OWA failure

Had a call about an older Exchange 2010 server this morning. Users were having problems working with Outlook Web App.

I saw a high volume of errors in that Application event log. This error was referring to both autodiscover and '/EWS/Exchange.asmx' and was appearing multiple times per minute.

Event ID: 3
Source: System.ServiceModel 3.0.0.0
Category: WebHost
WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/32001227
 Exception: System.ServiceModel.ServiceActivationException: The service '/Autodiscover/autodiscover.xml' cannot be activated due to an exception during compilation.  The exception message is: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
Parameter name: item. ---> System.ArgumentException: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
There was also this error about OWA.

Event ID: 108
Source: MSExchange OWA
Category: Configuration
Outlook Web App couldn't connect Exchange Web Services due to a configuration error. Response code = "null, webException.Status = SendFailure".
The errors in the log began at about 12:15am when someone had done a server reboot. So, something in that reboot triggered the issue. My first thought was updates, but no updates had been applied recently.

After a bit of research, it turned out to be the bindings for the default web site causing the issue. At some point, someone had added additonal http and https bindings to the default web site. When I removed the additional http and https bindings everything started properly and the errors no longer appeared.

I am guessing that someone added those bindings a long time ago, but they didn't start causing the issue until IIS was restarted as part of the reboot.

Tuesday, June 27, 2017

For Winnipeg IT Pros

For anyone from the Winnipeg IT Pros group reading the blog. Here are the PowerShell links that I promised to post.

PowerShell learning resources:
Slide deck from my presentation:

Friday, June 23, 2017

Errors on Public Folder Migration

As I was doing a public folder migration today, I got a couple of errors that took me some time to resolve. These are caused by mail enabled public folders migrated from Exchange 2003. You will see these errors when you run Get-MailPublicFolder on Exchange 2010. Some of these errors will show up in the public folder migration logs when migrating to Exchange 2016. So, I prefer to clean these up first to ensure migration is successful.


Error #1

WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Could not convert property OnPremisesObjectGuid to type Guid. Byte array for GUID must be exactly 16 bytes long.
My best guess is that this property is left over from Exchange 2003 (or maybe earlier). The quick fix is to disable mail for the public folder and then mail-enable it again. However, when you do so, verify the email addresses before and after.

Error #2


WARNING: The object domain.com/Microsoft Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Property expression "PF Name" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fit US-ASCII string in the e-mail address, which is generated from such an alias.

This error is most commonly caused by a space in the Alias property. Update this property to remove spaces and the error should be gone.

Saturday, June 17, 2017

Multiple Moderation Approval Requests

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4.

Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term).

Here is the process we saw:
  1. Message requiring moderation sent.
  2. Approval request sent to moderator.
  3. Moderator approves request
  4. Approval request sent to moderator
  5. Moderator approves request
  6. Repeat request and approval process a few more or a lot more times.
This process was happening even though we had not moved any mailboxes to Exchange 2016 yet. 

When searching, there were very few references to this issue on the Internet or support forums. However, there were a few suggestions that were consistent:
  • Ensure arbitration mailboxes are moved to Exchange 2016 (one of these stores the messages until they are approved).
  • Delete and recreate rules.
  • Move moderators mailbox to Exchange 2016.
  • Restart transport services.
All of these things were done but we still had issues. However, when both mailboxes were on Exchange 2016, the approvals on messages seemed to only happen twice. This was better than the random number from before.

I reviewed the message tracking logs for errors and didn't see any. In the logs, each time an approval was received, the message was released for delivery, but then promptly moderated again. However, the second attempt to approve worked.

All of my initial testing was done using inbound messages. So, I tried some scenarios with both mailboxes on Exchange 2016, and these were my results:
  • Inbound messages routing through Exchange 2010 first - Two approvals required
  • Outbound messages routing out through Exchange 2010 - One approval only
  • Inbound messages routing through Exchange 2016 only - One approval only
Based on these results, we can see that Exchange 2010 coexistence definitely plays a role, because when Exchange 2010 is not part of the inbound routing the issue doesn't occur. This at least provided confidence that after migration was complete the issue would not be persist.

The other item that needed to be addressed was the Exchange 2016 CU4. Microsoft releases updates in matched sets and Exchange 2016 CU4 was one step behind Exchange 2010 SP3 RU17. In the hope that having both at the same update level would fix it, we applied Exchange 2016 CU 5. We did the update to CU 5 but there was not change.

Final testing indicated that any message that entered the Exchange organization through Exchange 2010 was subject to double approval. This happened if the message came in from external or was generated by a mailbox in Exchange 2010. The location of the moderator mailbox did not make a difference.

So, to minimize the issue:
  • Move all inbound message routing to Exchange 2016 sooner rather than later. This includes Internet mail and applications that send messages to be moderated.
  • Move mailboxes that generate the most messages to be moderated first. Once the source is in Exchange 2016 the problem is mediated.
I should also note that there was a red herring in the application event log. We saw this error:
Event ID 1051, MSExchange Extensibility
Warning, MExRuntime
 Agent 'Approval Processing Agent' caused an unhandled exception 'SmtpResponseException: 250 2.1.5 APPROVAL.ApprovalRequestUpdated; approval request updated successfully' while handling event 'OnCreatedMessage'
However, review of the logs indicated that the error was present before the issue appeared. So it appears to be noise rather than useful information.

Thursday, June 15, 2017

Firefox - Exchange 2016 and NS_ERROR_NET_INADEQUATE_SECURITY

I'm working on an Exchange 2016 migration project and it was all looking good. We setup the load balancer and verified it worked for OWA and Outlook. Things were good.

Today we did the switch over and updated the DNS records to route all of the client traffic through Exchange 2016. IE and Edge were fine, but Firefox gave an error:
NS_ERROR_NET_INADEQUATE_SECURITY
Protocol issues for web browsers, certificates, and web servers can be tricky. It's hard to track it down with generic error messages like this. So, as an alternative to figuring out the details, I used IIS Crypto from Nartac with the best practices settings on the Exchange 2016 servers. After those were applied all was good. Just apply the settings and reboot.

For more about using the free IIS Crypto to see:
UPDATE: Here is an ongoing thread from the Exchange Technet forums. Seems to be known issue for Exchange 2016 at this point.