Sunday, August 13, 2017

Scripting Complexity vs. Simplicity

Note: Most of my blog posts are technical items that relate to performing a specific task or fixing a specific error. This one is more of an opinion piece. So, if my arguments sound reasonable, take the advice. If you disagree, by all means, go your own way.

I am a big believer that I need to understand the details of any script I run in a production environment. Unless that script is from Microsoft and provided to perform a specific task (such as migrating public folders), I will go through a script line by line to verify I understand it. This even applies to my own scripts. If I haven’t used a script for several months, I’ll review it before I use it again to make sure I know exactly what it’s doing.

I expect that most system administrators operate with the same requirement to understand the scripts they are running. At least I hope they do. I don’t want anyone blindly running a script I created without understanding the script and knowing what it will do in their environment.

So, from my perspective, running a script is very different from running a program/utility. A program is distributed as an executable and you simply need to trust the developer. I don’t need to trust a script writer if I can understand what the script is doing. The writer just saved me the effort of creating the script myself. It’s much faster to review and verify an existing script than it is to create one.

Simple is good, right?

If you need to understand a script, then simple is good. However, we often attempt want a script to be resilient to user error. That is, we want the script to do things like:
  • Not accept invalid values
  • Not make incorrect/invalid changes to objects
  • Warn when mistakes are about to be made
A more user-friendly script is good, but it becomes more complex and more difficult to interpret. So, does that mean complexity is good?

Know your audience

The balance of simple (but you need to know what your doing) and complex (with lots of error checking) depends on your audience. If the audience needs to understand the details of your script, then simplicity is better. If the audience does not care about the details inside the script then simplicity is less important. For example, if the help desk staff in your organization run standardized scripts to complete some specific tasks then user friendly is more important than simplicity. The help desk staff will be running the script without reviewing the contents or understanding how it works. So, if they give an invalid value, you need to account for that in your script. It’s worthwhile to make it user friendly to prevent errors in the production environment.

The customers I work with typically will look at my scripts before running them. So, these scripts need to be relatively short and understandable. The documentation that I provide for running these scripts also needs to be short and understandable. In this case simplicity is most important for the audience. For example, if there are several tasks that need to be accomplished such as updating email addresses, UPNs, etc. I will create separate scripts for each task. This makes each individual script easier to understand and document.

Striking a balance

I tend to work with customers that have varying knowledge levels and needs. Some of them would run a script without looking and others want a detailed understanding how the scripts work. I want my scripts to work in both scenarios. So, I need to put in some of the user-friendly bits like prompting for values when required and some basic checking for invalid values, but I don’t try to account for every possible error. To account for every possible error would be too much complexity.

Because I can’t (or won’t) put in tons of error checking, one thing I often do is display a confirmation on screen before performing an action. For example, I have a script to remove email addresses for a specified domain. Before script performs the action, it displays on screen the pattern being searched for and provides an email address from the first mailbox as an example of what is about to be removed. The script also displays the number of mailboxes that are being modified. Displaying this information allows the user to provide a final sanity check before approving removal from all mailboxes.

To make scripts more understandable, I use lots of comments that describe what each section of a script is doing.  This is useful for customers that want to review the script. It’s also useful for me when I want to review the script contents or modify the script. Embedding comment-based help in a script makes it easier to use, but detailed comments in the script are better for understanding how it works.

If you don’t get the right balance then your scripts won’t get used. That help desk person will think your script generates errors all the time and ignore it. The administrators you gave complex scripts to will build their own because they can’t understand yours.

Friday, June 30, 2017

June 2017 Security Update Breaks Outlook Search

The June 2017 security update for Windows causes problems for search in Outlook clients using cached mode.  Uses will see incomplete search results and may be notified that:
Search results may be incomplete because items are still being indexed.
This issue affects Windows 7, Windows 8.1, and Windows 10. It also affects all versions of Outlook.

Microsoft has fixed this issue with a new update released on June 27th. You can download the update here:
These fixes are for retail, OEM, and volume licensed versions of Outlook. If your Outlook is included with an Office 365 version of Microsoft Office, then updates are already available and should be installed automatically. If they are not installed automatically performed the following steps:
  1.  In Outlook, click the File tab, and click Office Account.
  2. Click Update Options and click Update Now.

To review Microsoft's documentation on this issue, see Issue #5 on this page:

Wednesday, June 28, 2017

Exchange 2010 OWA failure

Had a call about an older Exchange 2010 server this morning. Users were having problems working with Outlook Web App.

I saw a high volume of errors in that Application event log. This error was referring to both autodiscover and '/EWS/Exchange.asmx' and was appearing multiple times per minute.

Event ID: 3
Source: System.ServiceModel
Category: WebHost
WebHost failed to process a request.
 Sender Information: System.ServiceModel.ServiceHostingEnvironment+HostingManager/32001227
 Exception: System.ServiceModel.ServiceActivationException: The service '/Autodiscover/autodiscover.xml' cannot be activated due to an exception during compilation.  The exception message is: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
Parameter name: item. ---> System.ArgumentException: This collection already contains an address with scheme http.  There can be at most one address per scheme in this collection.
There was also this error about OWA.

Event ID: 108
Source: MSExchange OWA
Category: Configuration
Outlook Web App couldn't connect Exchange Web Services due to a configuration error. Response code = "null, webException.Status = SendFailure".
The errors in the log began at about 12:15am when someone had done a server reboot. So, something in that reboot triggered the issue. My first thought was updates, but no updates had been applied recently.

After a bit of research, it turned out to be the bindings for the default web site causing the issue. At some point, someone had added additonal http and https bindings to the default web site. When I removed the additional http and https bindings everything started properly and the errors no longer appeared.

I am guessing that someone added those bindings a long time ago, but they didn't start causing the issue until IIS was restarted as part of the reboot.

Tuesday, June 27, 2017

For Winnipeg IT Pros

For anyone from the Winnipeg IT Pros group reading the blog. Here are the PowerShell links that I promised to post.

PowerShell learning resources:
Slide deck from my presentation:

Friday, June 23, 2017

Errors on Public Folder Migration

As I was doing a public folder migration today, I got a couple of errors that took me some time to resolve. These are caused by mail enabled public folders migrated from Exchange 2003. You will see these errors when you run Get-MailPublicFolder on Exchange 2010. Some of these errors will show up in the public folder migration logs when migrating to Exchange 2016. So, I prefer to clean these up first to ensure migration is successful.

Error #1

WARNING: The object Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Could not convert property OnPremisesObjectGuid to type Guid. Byte array for GUID must be exactly 16 bytes long.
My best guess is that this property is left over from Exchange 2003 (or maybe earlier). The quick fix is to disable mail for the public folder and then mail-enable it again. However, when you do so, verify the email addresses before and after.

Error #2

WARNING: The object Exchange System Objects/PF Name has been corrupted, and it's in an inconsistent state. The following validation errors happened:
WARNING: Property expression "PF Name" isn't valid. Valid values are: Strings formed with characters from A to Z (uppercase or lowercase), digits from 0 to 9, !, #, $, %, &, ', *, +, -, /, =, ?, ^, _, `, {, |, } or ~. One or more periods may be embedded in an alias, but each period should be preceded and followed by at least one of the other characters. Unicode characters from U+00A1 to U+00FF are also valid in an alias, but they will be mapped to a best-fit US-ASCII string in the e-mail address, which is generated from such an alias.

This error is most commonly caused by a space in the Alias property. Update this property to remove spaces and the error should be gone.

Saturday, June 17, 2017

Multiple Moderation Approval Requests

I recently did a migration from Exchange 2010 to Exchange 2016 where the client uses a high volume of moderated messaging. There were over 100 transport rules that did message moderation of some sort. The initial deployment consisted of Exchange 2010 SP3 RU17 and Exchange 2016 CU4.

Deployment of Exchange 2016 into the Exchange 2010 environment didn't seem to have any effect. However, after we directed the internal namespace to Exchange 2016 for proxying, the approvals generated by the transport rules when whacky (yep that's the technical term).

Here is the process we saw:
  1. Message requiring moderation sent.
  2. Approval request sent to moderator.
  3. Moderator approves request
  4. Approval request sent to moderator
  5. Moderator approves request
  6. Repeat request and approval process a few more or a lot more times.
This process was happening even though we had not moved any mailboxes to Exchange 2016 yet. 

When searching, there were very few references to this issue on the Internet or support forums. However, there were a few suggestions that were consistent:
  • Ensure arbitration mailboxes are moved to Exchange 2016 (one of these stores the messages until they are approved).
  • Delete and recreate rules.
  • Move moderators mailbox to Exchange 2016.
  • Restart transport services.
All of these things were done but we still had issues. However, when both mailboxes were on Exchange 2016, the approvals on messages seemed to only happen twice. This was better than the random number from before.

I reviewed the message tracking logs for errors and didn't see any. In the logs, each time an approval was received, the message was released for delivery, but then promptly moderated again. However, the second attempt to approve worked.

All of my initial testing was done using inbound messages. So, I tried some scenarios with both mailboxes on Exchange 2016, and these were my results:
  • Inbound messages routing through Exchange 2010 first - Two approvals required
  • Outbound messages routing out through Exchange 2010 - One approval only
  • Inbound messages routing through Exchange 2016 only - One approval only
Based on these results, we can see that Exchange 2010 coexistence definitely plays a role, because when Exchange 2010 is not part of the inbound routing the issue doesn't occur. This at least provided confidence that after migration was complete the issue would not be persist.

The other item that needed to be addressed was the Exchange 2016 CU4. Microsoft releases updates in matched sets and Exchange 2016 CU4 was one step behind Exchange 2010 SP3 RU17. In the hope that having both at the same update level would fix it, we applied Exchange 2016 CU 5. We did the update to CU 5 but there was not change.

Final testing indicated that any message that entered the Exchange organization through Exchange 2010 was subject to double approval. This happened if the message came in from external or was generated by a mailbox in Exchange 2010. The location of the moderator mailbox did not make a difference.

So, to minimize the issue:
  • Move all inbound message routing to Exchange 2016 sooner rather than later. This includes Internet mail and applications that send messages to be moderated.
  • Move mailboxes that generate the most messages to be moderated first. Once the source is in Exchange 2016 the problem is mediated.
I should also note that there was a red herring in the application event log. We saw this error:
Event ID 1051, MSExchange Extensibility
Warning, MExRuntime
 Agent 'Approval Processing Agent' caused an unhandled exception 'SmtpResponseException: 250 2.1.5 APPROVAL.ApprovalRequestUpdated; approval request updated successfully' while handling event 'OnCreatedMessage'
However, review of the logs indicated that the error was present before the issue appeared. So it appears to be noise rather than useful information.

Thursday, June 15, 2017

Firefox - Exchange 2016 and NS_ERROR_NET_INADEQUATE_SECURITY

I'm working on an Exchange 2016 migration project and it was all looking good. We setup the load balancer and verified it worked for OWA and Outlook. Things were good.

Today we did the switch over and updated the DNS records to route all of the client traffic through Exchange 2016. IE and Edge were fine, but Firefox gave an error:
Protocol issues for web browsers, certificates, and web servers can be tricky. It's hard to track it down with generic error messages like this. So, as an alternative to figuring out the details, I used IIS Crypto from Nartac with the best practices settings on the Exchange 2016 servers. After those were applied all was good. Just apply the settings and reboot.

For more about using the free IIS Crypto to see:
UPDATE: Here is an ongoing thread from the Exchange Technet forums. Seems to be known issue for Exchange 2016 at this point.

Tuesday, June 6, 2017


Today while moving a mailbox from on-premises to Office 365 in a hybrid environment, I got the following error:
Transient error SourceMailboxAlreadyBeingMovedTransientException has occurred. The system will retry (5/620).
This error occurs when a previous move attempt did not get cleaned up properly. From a bit of reading, this should timeout and fix itself after about 2 hours. However, since I didn't want to wait that long, I did the following that got it going again.
  • IISReset.exe to restart the web services
  • Restart the Microsoft Exchange Mailbox Replication service
It is possible that only one of those two items was required, but I was more concerned about getting the move going than recording exact details.

Monday, May 29, 2017

Firefox Error Accessing Office 365

Microsoft is having a certificate trust issue today for Firefox when accessing Office 365 services. I haven't bothered digging into the details, but it's something to do with trust checking by using OCSP (Online Certificate Status Protocol).

The error you see is "SEC_ERROR_OCSP_INVALID_SIGNING_CERT" and looks like this:
Other browsers are not affected. So, you work around this by using Chrome, IE, or Edge.

If you prefer to continue using Firefox, you can modify the setting for certificate verification with the following steps:
  1. In the address bar, type about:config and press Enter.
  2. Click the I accept the risk button.
  3. Scroll down to security.ssl.enable_ocsp_stapling and double-click it to set the value to false.
Given that other browsers don't use this type of security, it's fairly safe to disable.

Friday, May 12, 2017

Everyone Dynamic Group with Office 365 Hybrid

Many organizations have configured a dynamic distribution group that includes all users with mailboxes. This Everyone group is used for sending out company notifications and the like. Because this group is dynamic, it's automatically updated as new mailboxes are added and removed.

When you implement hybrid mode with Office 365, the dynamic group will include on-premises mailboxes but not cloud mailboxes. This is because cloud mailboxes are a different recipient type that is not included. Cloud mailboxes are MailUser recipients (an AD user with an email address but no mailbox).

One way to fix this is to modify the dynamic distribution group to include MailUser recipients. The screenshot below has enabled Users with external e-mail addresses.

This will include users with cloud mailboxes. This is the recipient filter for the dynamic distribution group:
((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(Name -like 'SystemMailbox{*')) -and (-not(Name -like 'CAS_{*')) -and (-not(RecipientTypeDetailsValue -eq 'MailboxPlan')) -and (-not(RecipientTypeDetailsValue -eq 'DiscoveryMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'ArbitrationMailbox')))
Note that everything after the two recipient types is added automatically when you set the filter. So, don't include that information when you set a filter. If you do, those requirements will be in there twice.

Unfortunately, the recipient type UserMailbox includes on-premises resource mailboxes and the MailUser recipient type includes cloud resource mailboxes. To exclude those from our dynamic distribution group we need to manually configure the filter. We can do this by excluding the following values for RecipientTypeDetails:
  • RoomMailbox
  • EquipmentMailbox
  • RemoteRoomMailbox
  • RemoteEquipmentMailbox
I set the following filter to achieve this:
((((RecipientType -eq 'UserMailbox') -or (RecipientType -eq 'MailUser'))) -and (-not(RecipientTypeDetailsValue -eq 'RemoteRoomMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'RoomMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'RemoteEquipmentMailbox')) -and (-not(RecipientTypeDetailsValue -eq 'EquipmentMailbox')))
Now, the next challenge is allowing cloud users to see this dynamic distribution group. This is an issue because Azure AD Connect does not synchronize dynamic distribution groups to Office 365. For more information about this see:

Wednesday, May 10, 2017

Removing a Cloud Mailbox from A Synced User

The title for this post is a bit incomplete because there wasn't enough space for a full explanation. Here is the scenario I had...

Before setting up hybrid mode for an Exchange 2010 organization, a user account was created in Office 365 with the same UPN as a user account in on-premises. The on-premises domain had been added to Office 365, but directory synchronization was not yet in place.  As part of testing, the user account had been give an Office 365 license, which then created an online Exchange mailbox.

When directory synchronization was configured, the on-premises AD account was matched with the existing cloud user. However, the on-premises user has an on-premises mailbox and the cloud user has a cloud mailbox. When viewing the cloud user in Office 365, you could see the mailbox in the cloud.

Because there was an existing cloud mailbox I couldn't move the on-premises mailbox up to Office 365. There was also no way to remove the cloud mailbox to allow the on-premises mailbox to replace it. If the user license was removed, then the cloud mailbox was removed, but came back again when the license was re-added.

To fix this I had to delete the cloud user account and recreate it. To delete the cloud user account, I moved it into an organizational unit that Azure AD Connect was not syncing. Then I purged the user account in Office 365 with the following cmdlet:
Remove-MsolUser -UserPrincipalName -RemoveFromRecycleBin
Finally I moved the user back to an OU that was synchronized. This recreated the cloud user account without a mailbox and the account was identified as having an on-premises mailbox.

Monday, May 8, 2017

New Hybrid Free/Busy Fails Cloud to On-Prem

When you configure hybrid mode, calendar sharing is automatically configured between on-premises and cloud users. You don't need to configure anything. However, in some cases, you may need to perform an IISreset.

On a recent project implementing implementing hybrid mode for an Exchange 2010 organization, we enabled hybrid mode and tested connectivity. Everything looked good for mail flow and mailbox moves. However, free/busy lookups failed from the cloud users to on-premises. When we tried to view the on-premises users' calendars the status was "No Information".

When you're not sure what's going on, it's always a good idea to use the Remote Connectivity Analyzer to test things out. There is an Office 365 tab which includes a Free/Busy test. When I ran this test, it was successful, as shown in the figure below. However it still wasn't working.

It's been a while since I've implemented Hybrid for Exchange 2010. With a quick search, I found this blog post that reminded me that an IIS reset will fix this up:
Most Exchange IIS related issues resolve themselves after a period of time. That period of time might be 12-16 hours, but usually there is some background process that refreshes things.

In this case, hybrid mode had been configured for multiple days. The free/busy lookups required an IISreset or a server reboot to fix.

This fix is certainly relevant for Exchange Server 2010. It may also be relevant for later versions of Exchange Server.

Wednesday, April 26, 2017

Must Run O365 Hybrid Wizard with IE

Today I made the mistake of downloading and attempting to run the Office 365 Hybrid Deployment Wizard by using Firefox. When you do, the app starts to launch and the fails. When you click on the Details button to open the log file, you see this under the error summary:
Deployment and application do not have matching security zones.
Download the app from within IE and no problems.

Link to download the wizard:
UPDATE: Also worth noting that a similar error with the wizard not running occurs if the .application file extension is not associated with Internet Explorer.

Tuesday, April 18, 2017

Script to Synchronize Primary Email Address with UPN

When planning an Office 365 implementation, it is best practice to start by assuming that UPN for signing in to Office 365 should match the user email address. If you don't configure it this way, then users have two separate items (their UPN for signing in and their email address) that look very similar. In many cases users are confused by the similarity.

If you are synchronizing  your on-premises Active Directory with Office 365 (in most cases you do) then you need to set the UPN for the on-premises user accounts with the correct UPN. The UPN from on-premises user accounts is synchronized to Office 365 to create the ID for signing in.

Most organizations are not using the UPN on user accounts for authentication on-premises. The option has been there since Windows 2000, but most organizations still use the domainname\username format for authentication. However, you need to verify if any user accounts are using the UPN for authentication before making this change. At minimum, you should communicate with your application and system administrators to see if they are aware of anything that might use UPNs. If your organization has issued certificates to users, they might be using UPN as the unique identifier for the certificate.

The script below does the following:
  • Obtains a list of all users where the proxyAddresses attribute has a value. This is done so that the result include only user accounts with an Exchange attributes configured.
  • Identifies the primary email address based on the all caps "SMTP:" text.
  • Strips out the "SMTP:" text from the primary SMTP address.
  • If the new UPN and the existing UPN do not match the user account is updated and the change is logged.
The location and name of the log file are configured in $logfile. You need to manually configure this variable and verify that the necessary folders exist.

 #Log folder must already exist  
 $logfile = "C:\Scripts\SyncUPN.txt"  
 #Adds timestamp to log file  
 Get-Date | Out-File -FilePath $logfile -Append  
 #Obtains only users with valid proxyAddresses attribute   
 $users = Get-ADUser -Properties proxyAddresses -Filter {proxyAddresses -like "*"}  
 #Prepare variables for processing status  
 $total = $users.count  
 $current = 0  
 Foreach ($u in $users) {  
   #Find primary SMTP address for user  
   $primarySMTP = $u.proxyAddresses | Where-Object {$_ -clike "SMTP:*"}  
   #Remove "SMTP:" to create the new UPN value  
   $newUPN = $primarySMTP.Substring(5)  
   #Set the new UPN value only if required  
   If ($u.UserPrincipalName -ne $newUPN) {  
     $u.DistinguishedName + " Old UPN: " + $u.UserPrincipalName | Out-File -FilePath $logfile -Append  
     $u.DistinguishedName + " New UPN: " + $newUPN | Out-File -FilePath $logfile -Append  
     Set-ADUser $u -UserPrincipalName $newUPN      
   } #end if  
   #Processing status  
   $current += 1  
   Write-Progress -Activity "Processing users to update UPN to primary email address" -Status "Progress: $current" -PercentComplete ($current/$total*100)   
 } #end foreach  

Script to Remove Old Domains from User Email Addresses

When managing email addresses and domains in Exchange Server, old email addresses are never removed automatically. This is good because it ensures that email addresses on a mailbox are never accidentally lost. However, you may want to clean up old domains or address formats that are no longer in use.

Some common scenarios where you might want to remove an old domain:
  • An SMB deployment of Exchange Server where a .local domain was added as the first domain for email addresses.
  • Old GroupWise addresses are left in place from an older migration.
  • Obsolete domain left over from a company merger many years ago
I often find that obsolete domains are identified when I run IDFix as part of preparing to migrate to Office 365. To simplify the removal of obsolete domains, I have created the following script.

A few things to note:
  • You need to set $RemovePattern to identify the domain to be removed. Any email addresses matching this pattern will be removed from proxyAddresses attribute in Active Directory objects.
  • The script uses Get-ADObject rather than Get-ADUser to make sure that the domain is removed from distribution groups too.
  • This version of the script is capable of removing multiple instances of a matching email address. So, if a user has several email addresses in the old domain, all of them are removed.
  • At the end of the script, I use Write-Progress to display a status bar. It's not necessary, but if there is a large number of users it's nice to see activity on the screen instead of just waiting and hoping it's doing something.

 #This pattern is used to match the email addresses being removed.  
 #Test that this pattern finds the correct users and email addresses  
 #before running this script.  
 #Example: $pattern = "smtp:*"  
 $RemovePattern = "smtp:*@olddomain.local"  
 Import-Module ActiveDirectory #only required for 2008 R2  
 #Get the users that have an email address that matches the pattern  
 Write-Host "Querying objects...This may take a moment"  
 Write-Host ""  
 $objects = Get-ADObject -Filter {ProxyAddresses -like $RemovePattern} -Properties ProxyAddresses  
 #Identify address being removed from first user for warning  
 [String]$proxyexample = $objects[0].proxyaddresses -like $RemovePattern  
 #Display warning and get confirmation  
 Write-Host "You are going to remove email addresses that match the following pattern:"  
 Write-Host -ForegroundColor Red "$RemovePattern"  
 Write-Host ""  
 Write-Host "This is an example from the first object:"  
 Write-Host -ForegroundColor Red "$proxyexample"  
 Write-Host ""  
 Write-Host "This will modify $($objects.count) objects"  
 Write-Host ""  
 $confirm = Read-Host "Enter Y to continue"  
 If ($confirm -ne "Y") {Break}  
 #Prepare variables for processing status  
 $total = $objects.count  
 $current = 0  
 #Processing users to remove addresses  
 Foreach ($o in $objects) {  
   #Build list of addresses to remove for object  
   #required because there might be multiple that match  
   $proxy= New-Object System.Collections.ArrayList  
   Foreach ($a in ($o.ProxyAddresses)) {  
     If ($a -like $RemovePattern) {  
       $proxy.add($a) | Out-Null  
     } #end if  
   } #end foreach  
   #Remove each bad address  
   Foreach ($p in $proxy) {  
     Set-ADObject $o -Remove @{'proxyAddresses'=$p}  
   #Processing status  
   $current += 1  
   Write-Progress -Activity "Removing email addresses that match pattern" -Status "Progress: $current" -PercentComplete ($current/$total*100)   
 } #end foreach  

Thursday, April 13, 2017

Change All UPNs in a Domain

I needed to update all UPNs in a domain today. It was pretty quick to figure out, but here is one line to take care of it for you.

Get-ADUser -Filter * | ForEach-Object { Set-ADUser $_ -UserPrincipalName ($_.UserPrincipalName).Replace("OldDomain","NewDomain")}
Remember to make the pattern in the OldDomain unique enough that you don't accidentally change things you don't intend to. For example, if you are changing from a .local domain in the UPN to a .com, make sure that you replace ".local" and not "local" on the off chance one of the user IDs includes "local" in the name.

If there are any user accounts without a UPN, then an error is generated for those accounts. My domain had 4 accounts without a UPN:
  • krbtgt - default account used for kerberos
  • IWAM_ServerName - Old IIS account from Windows 2003
  • IUSR_ServerName - Old IIS account from Windows 2003
  • support_XXXXXXX - Used by Help and Support service

Suppress Results when Adding Items to an ArrayList

I ran into a mildly annoying feature when adding items to an array list when using PowerShell today. An array list is an expandable array of items with better performance than a normal array when working with large data sets.

Each time I added an item to the array list, it echoed back the index number of the list item. When I added the first item in the list, the number 0 was displayed on the screen. Adding a second item would echo back the number 1. For example:

I would prefer my script to be silent when running except when there is data that I want to display. However, there is no option obviously available for that purpose. Instead, you need to redirect the output to $null. There are a few ways to do this and any one will work:
$proxylist.add($a) > $null
$proxylist.add($a) | Out-Null

Sunday, April 2, 2017

Dell Open Manage System Administrator Hangs (or Unavailable)

Just ran into an issue on Dell servers using the Dell Open Manage System Administrator software. This software runs on the server to let you see hardware details such as failed components and RAID configuration.

My first issue was when running the System Administrator icon from the desktop. This icon opens up and web page to access System Administrator. However, when Internet Explorer was launched, it came up with the error:
This page can't be displayed
So, I did the standard stuff:
  • restart services
  • verify DNS resolution
  • verify port 1311 is not blocked by firewalls and is listening
Everything looked good, but it wasn't working. One person on a discussion group indicated that they found it was because the older versions of System Administrator used older encryption algorithms for TLS and so the browser was blocking connectivity.

I attempted to resolve it first by updating the existing installation of Server Administrator. This changed the problem to hanging while trying to access the app, but didn't fix it.

The final fix was to remove older versions of System Administrator and install the latest version fresh. It seems that upgrading kept some older incorrect settings. The new install wiped out the older settings and all was good.

So, if Server Administrator is reporting "This page can't be displayed" or hanging when you attempt to access it, try an uninstall and reinstall. You don't need to reboot.

Tuesday, March 14, 2017

Making Sense of Office 365 Plans

If you're just starting to look at Office 365 as a solution for your organization, the various plans can be overwhelming and confusing. I'm going to try and boil down all of the Office 365 plan information to just the essentials that allow you to make an informed decision.

This is all based on research done March 2017 and the prices I include are Canadian dollars. You should verify that these features and prices are still correct for your scenario before making any decisions. I've includes some links at the bottom of this article to Microsoft documentation for you to verify. Microsoft should be keeping that content up to date.

I'm going to focus on Office 365 plans for small business and enterprise. However, whether you are small business, non-profit, enterprise, or education, there are basically three generic Office 365 plans available:
  • Office 365 desktop apps (Word, Excel, Outlook, etc)
  • Cloud services (Exchange, Skype for Business, etc)
  • Office 365 desktop apps and cloud services
Most of the organizations I work with are looking for the cloud services. The initial driver most of them have is replacing an older installation of Exchange Server. At the same time, they can evaluate whether including Office 365 desktop apps is appropriate. I do not have any customers subscribing to only the Office 365 desktop apps.

The Office 365 plans for small business (300 user max) are:
  • Office 365 Business (desktop apps)
  • Office 365 Business Essentials (cloud services)
  • Office 365 Business Premium (Business + Business Essentials)
The Office 365 plans for enterprise (unlimited users) are:
  • Office 365 ProPlus (desktop apps)
  • Office 365 Enterprise E1 (cloud services)
  • Office 365 Enterprise E3 (ProPlus + E1 + a few cloud features)
  • Office 365 Enterprise E5 (E3 + cloud telephony)
It is possible to continue using your existing OEM, retail, or volume licensed edition of Microsoft Office with Office 365 cloud services. So, if you recently purchased 100 volume licenses of Office 2016, that is not a lost investment. You can use those licenses until you are ready to upgrade to a newer edition of Office and then evaluate whether you prefer to purchase new volume licenses for Microsoft Office or change your Office 365 licensing to include the desktop apps.

Office 365 Desktop Apps

The Office 365 desktop apps are similar to the Microsoft Office Suite that you can buy retail, OEM, or through volume licensing. The biggest difference you'll notice is that these apps are streamed to desktops from Office 365 rather than a traditional installation (however it looks the same from a user perspective). This means that they are automatically updated outside of the Windows Update process. This should make the apps more secure because they will be updated faster than most organizations typically deploy updates. However, you do loose control over the update process and this may be a concern in organizations with specialized plugins.

The licensing for the Office 365 desktop apps is per named user rather than per computer. Each user can have up to five instances of the Office 365 desktop apps on devices. This allows a single user to put the Office 365 desktop apps on a work computer, a work laptop and a home computer. However, this does not mean that an organization with 20 users and 20 computers should purchase just 4 user licenses and install the Office 365 desktop apps 5 times per license. You need to license the Office 365 desktop apps for each user.

Licensing for Office 365 desktop apps is verified by signing in to Office 365. On each computer with the Office 365 desktop apps, you need to sign in to Office 365 at least every thirty days to verify that the license is still valid. This is a concern only in scenarios where a mobile computer would not have Internet access for more than 30 days.

Office 365 Business and Office 365 ProPlus contain the same apps:
  • Outlook
  • Word
  • Excel
  • PowerPoint
  • OneNote
  • Access
However, there are minor differences in app functionality. The following features are available only in Office 365 ProPlus and are not available in Office 365 Business:
  • Outlook:
    • Information Rights Management (IRM)
    • Data Loss Prevention (DLP)
  • Access:
    • Database Compare
  • Excel:
    • Spreadsheet Compare
    • Spreadsheet Inquire
    • Power Map
    • Power Pivot
    • Power Query
    • Power View
  • Support for Group Policy-based configuration
  • Support for Office add-ins, ActiveX, and browser helper objects (BHO)
  • Roaming settings
For a complete comparison of features, see Office Applications Service Description.

Note that some older documentation may reference that:
  • "Access is not included in Office 365 Business." Update: Access is included with Office 365 Business starting in November 2016.
  • "Outlook in Office 365 Business cannot access Exchange in-place archives." Update: The current version of Outlook in Office 365 Business can access in-place archives (also referred to as archive mailboxes). See Outlook license requirements for Exchange features.

Office 365 Plans with Cloud Services

Most cloud services in the small business and enterprise plans are the same. All of the small business and enterprise plans include the following:
  • Mailbox and calendar
  • Office Online apps - web-based versions of Word, Excel, and PowerPoint
  • OneDrive - personal file storage
  • SharePoint Online - shared file storage
  • Skype for Business - teleconferencing and instant messaging
  • Active Directory integration - synchronizes Active Directory users into Office 365
  • Yammer - Group discussions
The Office Online apps are very useful for performing quick edits to documents stored in OneDrive or viewing email attachments. In most cases, users prefer to continue using standard Microsoft Office desktop apps. However, in a very cost conscious organization, with limited needs, the online Office apps may be sufficient.

The graphic below summarizes some of the similarities and differences between the small business and enterprise plans:

Some differences to highlight are:
  • The small business plans are limited to 300 users. However, you can have a mix if small business and enterprise licenses in a single Office 365 tenant.
  • The small business and E1 plans have 50 GB mailboxes with 50 GB archives. The E3 plan has a 100 GB mailbox with unlimited archives. For small business and E1 plans, you can purchase an Archiving add-on for unlimited archiving.
  • Only the E3 plan supports litigation hold and data loss prevention for email.
  • The small business and E1 plans have 1 TB of OneDrive storage per user. The E3 plan has 5 TB of OneDrive storage per user.
  • SharePoint Online has 1TB of storage per Office 365 tenant and then 500 MB additional storage per licensed user. Storage consumed by Office 365 Teams come out of this pool.
  • All plans include Skype for Business, but only enterprise plans can add unified communications.
  • Only enterprise plans have meeting broadcast that allow presentations to thousands of users.
  • Only the E3 plan supports Azure Rights Management to encrypt and secure files.

Why Wouldn't I use Office 365?

The main reason you might not be able to use Office 365 is compliance and recovery purposes. For example, you can recover deleted items in Exchange Online for up to 30 days (only 14 days by default). There is no option to recover deleted data older than that.

It's possible for you to work around this issue, but it's not inexpensive. You will need to implement some sort of third-party backup or archiving solution. However, you'll need that type of system if you have Exchange on-premises anyway. It just becomes more complicated to backup data in the cloud. That said, third-party vendors have recognized this need and more backup and compliance products for Office 365 are being made available.


Most smaller organizations do not need the extra features included in the enterprise plans. The differences Business Essentials and E1 are mostly whether you can add on other features. The core level of functionality is mostly the same. If you have less than 300 users the Business Essentials and Business Premium plans are what you should evaluate first due to the cost savings.

For a very small organization, of 5 or 10 users, it's a pretty easy decision to use Office 365 based only on avoiding the cost of the local Exchange Server and hardware. But, you also avoid other costs like backup software, anti-spam software, and anti-virus software for a local Exchange server.

For slightly larger organizations, you might do a cost comparison and see that the cost of on-premises Exchange is about the same as Office 365 licensing. However, Office 365 is giving you high availability across multiple data centers that you probably can't implement your self. Then throw in the ability to have large mailboxes (up to 50 GB), that most on-premises instances of Exchange don't allow, and Office 365 is a winner on features.

For even larger organizations, you might find that Office 365 licensing is more expensive than purchasing and managing on-premises Exchange. However, Office 365 is more than just email. There are additional features like Skype for Business, OneDrive, and SharePoint Online. So, while you may start evaluating Office 365 as a replacement for on-premises email, remember about the extra value the additional services provide and identify whether those services are useful for your organization. Maybe having video conferencing with Skype for Business is a big value add in your organization.

If you have more than 300 users, you can mix business and enterprise plans in the same Office 365 tenant. However, you probably want to be consistent and stick with the enterprise plans to avoid user and helpdesk confusion. Imagine that you implement a Group Policy object for managing Office 365 Proplus but half of your users are using Office 365 Business so that the GPO doesn't apply. It would be a mess.

If your organization is academic or non-profit, check out the Office 365 licensing available specifically to your type of organization. There are academic and non-profit licenses equivalent to business and enterprise plans. At time of writing the plans with only cloud services were free. The plans that include Microsoft Office apps are heavily discounted compared to business and enterprise plans. It almost becomes foolish to keep running your own internal Exchange server.

Useful Links

The following are some of the links I found useful:

Thursday, March 9, 2017

Exchange 2010 SP3 Hub Transport Upgrade Error

Ran into a new issue yesterday related to installing Exchange 2010 SP3. I was called in to help when the initial upgrade attempt failed. The error during SP3 installation was:
An unexpected error occured while modifying the forms authentication settings for path /LM/W3SVC/1. The error returned was 5506.
A screenshot of the error is below:

Doing a search didn't come up with much, but it did give this:
That link seemed to indicate that it could be related to the SSL binding on the default web site in IIS. Taking a look at the SSL binding, it seemed to be missing the certificate assignment. However, when I tried to add the certificate I got a strange error about the session be closed.

Ok then, since you won't let me add the SSL certificate to the binding by using IIS Manager, let's try with Exchange Admin Console. When I assigned the IIS service to the certificate in EAC, it all looked fine. I also took this moment to review the certificate and verify that the SAN names were correct. I also noted that it did indicate that there was a private key for the certificate.

After this the binding worked because we could access https://servername/owa URL, but it returned a 503 error. However, rather than attempting to fix that error, we tried the SP3 install again. Since a service pack upgrade rewrites a lot of the content in the IIS virtual directories we through we might get lucky and it would fix and configuration errors that we had.

During the next install, the installation of the Hub Transport role completed successfully, but now we got an error on the Client Access role installation. As we were actively troubleshooting I didn't write it down at the time, but it was something like:
Could not grant Network Service access to the certificate with thumbprint BIGLONGHEXTHUMBPRINT because...
Based on this I decided to review the certificate in the Certificates MMC snap-in. Again, all the details looked right. Maybe I can add the necessary permissions myself for Network Service. To access the permissions for a certificate, you right-click it, point to All Tasks, and click Manage Private Keys. This normally brings up a security dialog box. However for me it brought up the following error:
Object not found.
I interpreted this error to mean that either the private keys were not really present for the certificate. Or the Domain Admin account that we were using to access the certificate and run the install didn't have permission to access the private keys. In either case, since our Domain Admin account couldn't set permissions in the certificate, we were dead in the water.

Fortunately certificates are much less expensive than they used to be and we quickly obtained a new certificate with all of the necessary names from NameCheap. They might not have the best management tools for certificates, but the price is right. So, if this didn't work it didn't waste a lot of money.

After installing the new certificate and assigning the correct services to it, we ran the Exchange 2010 SP3 upgrade again. And after some nervous waiting, the upgrade completed properly. And the upgrade fixed all of the errors for the web services. Email for phones began to work immediately, as did OWA.

So, You Wanna Be a Computer Geek?

I recently did a presentation for an Introduction to Management Information Systems class at the University of Manitoba Asper School of Business.  Students in this class are just starting to look at how IT and business are interrelated. One of the students asked me for advice on getting into the IT industry and this content grew out of that.

I suppose the more polite way to phrase it would be:
  • So, you'd like to work in the IT industry?

Areas of IT

One of the things that surprises many people looking at IT is that wide range of job roles. When you haven't been working in the industry, you tend to think that there is just the one role of computer geek. And your impression is likely that the computer geek does all computer related stuff including physically repairing computers.

In actuality, there are multiple job roles in IT. And, the more you learn about IT, the more you realize that you understand only your little corner of the world. The more you learn, the more you realize how little you actually know. Don't be disheartened as you go through that process. Nobody knows all of it.

Some of the job roles  are:
  • Help Desk - Takes support calls from users when computers or applications are not working correctly.
  • Desktop Support - Manages desktop computers which includes software deployment, repairing software problems, and repairing hardware problems.
  • Server/System Administration - Responsible for implementing and maintaining servers. This includes the server hardware and operating systems, Active Directory, and potentially some additional software that runs on servers such as SQL server.
  • Application Support - Responsible for configuring and maintaining specific business applications. For advanced troubleshooting, they act as an interface for interacting with the application vendor for support.
  • Database Administrator - A specialist that is responsible for managing and maintaining databases that are used by applications. This role troubleshoots database performance issues and implements the requirements specified for individual applications.
  • Network Administrator - Responsible for configuring switches, routers, firewalls, and other network specific devices.
  • Programmer - Builds and maintains customized software used internally. Programmers can also perform customizations for off-the-shelf software. Web development is also in this category.
  • System/Business Analyst - Responsible for helping bridge the gap between business units and the technical side by translating business requirements into technical requirements that can be implemented.
  • System Architect/Designer - This role is responsible for understanding how systems work and a high level and ensuring that any new applications/solutions work within the framework already developed for existing systems.
It's important to realize that not every organizations has all of these roles. Smaller organizations tend to combine these roles together. For example a small business may have 1 or 2 IT staff that effectively fill all of these roles.

If you want get into IT, you need to understand which role you're hoping to fill. The education requirements and career progression for each role is different.

Educational Requirements

When I started in this industry in the 1990s, many of us were self-taught and didn't have any formal computer training. That is not typical today. In most cases, you need to have formal related training in order to be considered for a position.

Help desk and desktop support are often thought of as entry level positions. The education requirement for these roles is usually a one or two year program that includes content on configuring desktop computers and some information about managing servers.

In larger organizations, desktop support can be an area of specialization rather than just a starting point. There is opportunity to move up within desktop support and have a wide scope of responsibility. For example, a large organization can have specialists that develop processes for deploying operating systems, applications, and configuring computers centrally.

Server/System administrators typically require at minimum the same one or two year program that is required for help desk and desktop support. However, this role is not entry level and you do require experience to obtain it. That on the job experience allows you to understand how all of the pieces really fit together and learn more technical details. In this role, you often have additional specialized technical training focused on specific products. Some organizations prefer a computer science degree for this role.

Application support can require a wide variety of technical skills. Depending on the organization, it may require a computer science degree or business degree. There will also be some element of training in the specific applications being supported. Some common applications such as Exchange Server for email may be taught as part of a formal education process. Other less common applications may be learned on the job or in training provided by the vendor.

Database administrator is a specialized role that requires specific education in database management. This can be a one or two year program or a computer science degree. There may also be training in how to use specific types of databases such as Microsoft SQL Server, Oracle, or MySQL.

Network administrators require specific training in how to configure network equipment. The most common way to show your knowledge in networking is to obtain industry certification from Cisco. Even if you don't use Cisco equipment in the job, having that certification shows you understand the general concepts that are required. Then you figure out the specific commands to implement what you need on equipment from a different vendor. Training for Cisco certification is provided in many one or two year technical courses.

You can get the training to be a programmer from technical colleges (2 year programs) or as part of a computer science degree. Generally speaking, a computer science degree will provide more theoretical knowledge that will help you advance more into design. A shorter program from a technical college will teach you programming, but less of the design aspects.

System/Business analyst is usually someone with broad business education and some technical knowledge. Often people in this role have a business degree with additional education or experience on the technical side.

A system architect/designer needs to have a broad range of technical experience, and years of it. In terms of formal education, it may be a business degree, computer science degree, or even an MBA. However, the real key here is that this is not an entry level position, it's something you work up to.

Industry Certifications

When you need to prove your knowledge of specific technologies, you'll most often end up obtaining industry certifications. Industry certification are exam-based certifications designed by the product vendors. I previously mentioned Cisco certification for networking, but many vendors offer certification for their products.

You do not need formal training in order to obtain most certifications. You can study on your own and then write the exam. Or, you can take short courses (often a week or less, but crazy expensive) that focus on the specific content related to that certification before writing the exam. Exams are available at testing centers throughout the world. Some certifications consist of multiple exams.

Here is information about some vendor certifications:
Some of the entry level certifications are included as part of formal training in technical schools. For example, you may get Cisco Certified Network Administrator (CCNA) or Microsoft Certified Professional (MCP) training. Another commonly included entry level certification is A+ certification for basic hardware and software configuration.

How Do I Decide?

If you're not already in the IT industry, it's pretty hard to figure out what you might want to do. I'm a firm believer in trying stuff out (or at least learning about it) to get a better understanding. It would be unfortunate to take a two-year programming course and then realize that you don't like programming at all.

The Internet is full of many resources on the technical details of help desk, desktop support, server administration, programming, and database administration. However, you may find it easier to start  learning about working in these roles by using content with some structure. Fortunately there is lots of that available for free on the Internet too.

The following resources are Microsoft-based because that's what I work with the most. There are many other worthwhile resources, but these are the ones I'm familiar with.

  • Microsoft Virtual Academy – Free online video training. This is no cost and Microsoft does it to spread knowledge about how to use their products. The IT Pros content is what I deal with, but you can also check out the developer (programmer) and data pro (database) content.
  • Channel 9 – Free online videos (typically 1 hour or less) about Microsoft products and features. Presentations from Microsoft conferences such as Microsoft Ignite are also hosted here (in the events section). Many people attend these conferences (at a cost of several thousand dollars), but I find it hard to justify when I can view the same information the day after for free.
  • TechNet Virtual Labs – Hands-on virtual labs that give you experience actually working with Microsoft products. Want to try out using Windows Server and creating SQL databases? This gives you access to virtual machines running that software completely free of charge. No need to setup your own test lab when they provide it for you. The labs includes specific activities for you to try or do your own thing.
  • Free eBooks from Microsoft Press - Most of these books tend to be introductory, almost marketing level content. They do a good job of describing features without some of the technical details. This makes them good for getting an overview of the products as someone looking at the industry for the first time.

Add Your Own Comments

If you have any additional suggestions for this content, please leave a comment below. This was written up in a couple of hours and I'm sure there are important and useful items that I've missed.

Thursday, February 2, 2017

Site Mailboxes Deprecated in SharePoint Online

Just saw a notification in my Office 365 portal that site mailboxes are being removed from SharePoint online. Existing site mailboxes will continue to function for now, but after March 2017 you cannot create new site mailboxes.

It is recommended that you use Office 365 groups for collaboration instead. An Office 365 group behaves like a combination of a distribution group and shared mailbox combined with storage in SharePoint. It's a more complete collaboration solution, but you can use just the features that you want.

In September 2017, a process will begin to transition site mailboxes to Office 365 groups.

Here is a link with more info about Office 365 groups:

Tuesday, January 31, 2017

Windows 2003 Documentation in PDF

Someone at Microsoft must have decided it was time to clean up the support documentation. You'll now find that if you try to use a link referring to older Windows Server 2003 documentation or support docs, you instead get prompted to download a PDF. At first I didn't think much about this as I didn't really need the documentation.

Today I wanted to confirm some processes in a forest recovery (a low likelihood issue, but I'm doing up some documentation). Ok, I'll download this and find the content I want.

It turns out that this PDF is 150MB and 28000 pages. I'm sure it's complete, but not very convenient.

If anyone else is looking for the forest recovery info, it's on page 3078.

I should also point out that this content is relevant all the way up to Windows Server 2012 R2 (and I assume Windows Server 2016 also). MS has no other official forest recovery info that I've run across.

And just for fun, here's a link to the Windows 2003/2003 R2 retired content:

Saturday, January 28, 2017

PowerShell Script for Math Homework

My daughter needs to practice her multiplication tables. So, I came up with a little script that can help.

You can use the script on any Windows computer. Copy the code below into a text file and then name that file something like multiply.ps1. The file needs to end in .ps1 for Windows to recognize it as PowerShell.

You may also need to allow PowerShell scripts on your computer. Open a PowerShell prompt and run Set-ExecutionPolicy RemoteSigned.

If you have the file saved on your desktop, right-click it and select Run with Windows PowerShell

 $questions = Read-Host "How many questions?"   
  For($i=1;$i -le $questions;$i++) {   
   $first = Get-Random -Minimum 0 -Maximum 10   
   $second = Get-Random -Minimum 0 -Maximum 10   
   $answer = $first * $second   
   Do {  
     Write-Host "$first x $second = ??"   
     $response = Read-Host "Enter your answer"   
     If ($response -eq $answer) {   
          Write-Host "That is correct!"  
     Else {   
          Write-Host "Try Again"  
   Until ($response -eq $answer)  
 Write-Host "Well done! $questions questions completed!"

Friday, January 27, 2017

Full Restore for DC with NetBackup

I was doing some disaster recovery testing for Windows 2008 R2 domain controllers today with Veritas NetBackup. I’m running through and documenting some scenarios in a test environment. Better to document the steps before you need them!

Doing a non-authoritative and an authoritative restore went well by restoring the system state. Next up on my list was a full server restore.

The documentation for a full server restore was (to be kind) a bit fuzzy. The best of their articles I could find was this one:
At a high level, the instructions are:
  • Install and OS with the NetBackup client software.
  • Restore the drives (and don’t reboot yet)
  • Restore the system state
  • Reboot
My problem was that after the reboot I got a blue screen. After stopping the blue screen long enough to see the error, I saw this:
STOP: c00002e2 Directory Services could not start because of the following error:
The specified procedure could not be found
Error status: 0xc000007a
I did some searching and found lots of references to a corrupt AD database and fixing it by removing log files or doing a manual repair on the ntds.dit file. Just for kicks, I did try these because they were fast and easy, but not the answer.

This link from Microsoft gave me the hint I needed:
This link indicates that the error occurs when the Active Directory Domain Services role is removed before a domain controller is demoted. Basically, you have lobotomized DC that doesn’t have all the files anymore but is still trying to run the services. I tried to run ntdsutil and the file wasn’t there. That was a good hint that some files for AD DS were not there.

To fix my process, I installed the AD DS and DNS server roles before I did the restore. By doing those, all was good.

Tuesday, January 17, 2017

O365: Unable to Create Distribution Group

Microsoft is aggressively encouraging Office 365 customers to use Office 365 groups instead of traditional distribution groups. In the Exchange admin center, when you select to create a distribution group, you get a popup to create an Office 365 group instead, as shown below.

There is one difference between this popup and if you actually selected an Office 365 group. This window has an option to create a distribution list. You can see it in the screenshot above by the red arrow. I'm pointing out that option because I didn't see it at first and was only made aware of it by Microsoft.

I should also note that another work around is to create a distribution list in the Office 365 admin center. That option is still available and is the same as creating a distribution group in Exchange admin center.

Sunday, January 15, 2017

Office 365 Tech Support is Good!

As a technology professional, I dread calling tech support sometimes. Most of the time when you contact tech support (for any software), you get a front line person that is not terribly knowledgeable or useful. That first level person has access to a knowledgebase that is similar to what you could find by searching online. When that person can't help, they pass you up to a higher level of support that can likely fix your issue.

The other problem with most tech support is timeliness. You are often kept on hold for an extended period of time or are forced to contact support via email or web form and hope that they get back to you within a few hours. It's almost never quick.

My experience with Office 365 support today was amazing. I had a question on Sunday morning at about 11am and had an answer within 10 minutes. Here is what it looked like....
  1. I'm working on some labs and find that in the Exchange admin center, when I attempt to create a distribution group it actually prompts me to create an Office 365 group instead. I confirm this is the case in my own personal Office 365 tenant and a test tenant I'm working with for lab development.
  2. In the Office 365 admin center, in Support, I selected Let us call you. This option is not available until you at least attempt to search for a resolution to your problem.
  3. The Let us call you option showed an estimated wait time of 10 minutes. So, I entered my phone number and waited.
  4. Within about 5 minutes, I got a call from a very helpful person at Office 365 Support (thank you Bel).
  5. She listened to my concern and did a remote view on my system to confirm the issue and identified it as a bug. She offered a work around of creating the distribution list in the Office 365 admin center (which does work) instead of the Exchange admin center.
  6. She also followed up with an email that stated she confirmed the issue in her own test environment and has reported it as a bug in the Exchange admin center user interface. Nice to know that there is a process in place to take care this rather than just giving me the work around.
Here is what was awesome:
  • Support was fast and I knew about how long it would take to be contacted. Sometimes the ambiguity of dealing tech support is the worst part. And, this was Sunday morning, not business hours.
  • The support person wasn't working from a script. She listened to my issue and then wanted to confirm it by remote viewing. There wasn't a long process of "well, let's try this...." I was not treated like a dummy as most tech support does.
  • This level of support is available to anyone. I don't have any special support contract. In fact the tenant for my email that I used to send the support request costs only about $12 per month. That's awesome support for a low cost product.
Update: Since this post, I've learned from Microsoft that the UI change for creating distribution groups in Exchange admin center is not a bug, but a design change. For details, see my other post here:

Exchange VM Hangs During Updates

I haven't run into this yet, but it appears that in some cases, Hyper-V virtual machines running Exchange Server will hang when installing updates. Specifically this seems to occur when running updates for Hyper-V integration services.

There are reports of KB3037623 specifically causing this issue.
The fix is to:
  1. Disable the Exchange services
  2. Apply the update
  3. Reenable the Exchange services
This blog posting provides detailed steps:

Tuesday, January 10, 2017

PowerShell Learning Resources

I'm doing some onsite PowerShell training this week and realized that I mention lots of resources but haven't provided a list of them anywhere for easy access. So, this posting is my best summary of Windows PowerShell related learning content from Microsoft. There are also a bunch of my links to my blog articles that I use as examples in class.

General Resources

Microsoft makes a lot of content available online for free. Here is a high level list:

Windows PowerShell Resources

Here are some resources specifically related to PowerShell:

PowerShell Examples

The following are examples of using PowerShell from my blog. They may or may not be useful for your purposes. I use them in class as examples that we review.

Wednesday, January 4, 2017

Finding Stale SIDs on GPOs

One of my clients has a tool from Microsoft that scans the AD infrastructure and generates a report of items that can fixed/improved. One of the items on a recent report was stale SIDs on GPOs that could affect GPO processing. However, the tools didn't give us the stales SIDs. Just said we had them.

First, let's talk about what a stale SID is...

All Windows security is based on a Security Identifier (SID) that is unique for each user or group. In the Access Control List (ACL) for an resource, it is the SID that is assigned permissions, not the name of a user or group. The Windows tools just translate that SID back to a user or group name for use to manage them easier.

A stale SID occurs when a user or group has been assigned permissions to access a resource and the user or group is later deleted. There is no link back from the user or group to where the permissions have been assigned. So, Windows cannot go back and remove the SID from the ACL. The SID that's left behind without a matching user or group object is a stale SID. When you are using graphical tools to view permissions and it shows a SID instead of a user or group name, that's typically a stale SID.

NOTE: Just because a graphical tool is showing a SID does not 100% guaranteed that the SID is stale. It could be a user or group from a trusted domain that the tool is having trouble resolving. If you have trusted forests or domains, you should verify that SID is in your domain.

If there were only a few GPOs, it would be fairly fast to use the Group Policy Management Console to find the stale SIDs. However, this client had about 500 GPOs and manually verifying the permissions would have been quite painful.

To find the stale SIDs on GPOs, I wrote up a small script that scans the GPOs and finds any security permissions that are unknown:

 Import-Module GroupPolicy  
 $gpo = Get-GPO -All  
 Foreach ($g in $gpo) {  
   $permissions = $g.getsecurityinfo()  
   Foreach ($p in $permissions) {  
     If ($p.Trustee.SidType -eq "unknown") {  
       Write-Host "Policy with unknown SID: $($g.DisplayName)"  
       Write-Host "Trustee SID: $($p.Trustee.Sid)"  
     } #end if  
   } #end foreach permissions  
 } #end foreach gpo  

Here is what the script does:
  • Loads the GroupPolicy module (required for Windows Server 2008 R2, Windows Server 2012 will do that automatically.
  • Pulls all GPOs into the variable $gpo.
  • Starts a foreach loop to process each gpo in $gpo.
  • Pulls the permissions for the current GPO into the $permissions variable by suing the getsecurityinfo() function for gpo objects.
  • Starts a foreach loop to process each permission in $permissions.
  • Tests whether the SidType for the trustee in the permissions is unknown. An unknown SidType identifies a SID that couldn't be resolved to a user or group.
  • The name of the gpo and the SID of the trustee are written to screen.
 This script writes output to screen, but you could easily modify it to dump the output to fine instead.