Office 365 Project Status and Issues
Phase 2 is complete (with that definition being all mailboxes identified by the affected WSU areas have been successfully migrated to Office 365!).
Over the next few days, we will be reviewing the remaining 2000ish mailboxes remaining on the Exchange Server. Many are old test and admin accounts, or accounts from separated employees. We will review and delete all the obvious system accounts and and employee accounts where the separation occurred over 6 months ago, and then we will migrate all the remaining accounts. This will maintain accounts for anyone who was inadvertently left off the lists and we haven’t caught it at this point. Any accounts that are not used will be automatically removed when Grouper comes online in the Fall.
Phase 3 is now starting. This is the migration of all of the rest of the WSU independent or department run mail systems to Office 365, and the consolidation of all mail accounts to one entity, that being NID@wsu.edu.
ITS will be contacting areas and working with them individually over the next few months to ensure we can meet area business outcomes. We will start with Tri-Cities and the ITS Unix Mail System this month (July 2017).
Direct all questions to the Project Manager Bill Rivers 5-0015 or email@example.com.
Thanks for everyone’s support on this WSU initiative.
With one more day to go for our scheduled migrations, we have finished migrating 98% of the Area mailboxes for People, Places and Things. Tonight we start the final cut-over for the remaining two Areas. Thanks to all you Area and Departmental support people for your cooperation on this. We realize much of the client support burden fell on your shoulders.
We have a bunch of unclaimed mailboxes that have not migrated and we are still sorting out the best method for handling these. In the meantime, if you find someone in your Area whose mailbox hasn’t moved yet, send an E-Mail to firstname.lastname@example.org.
- We have a week and a half to go. We have finished migrating 87.5% of the Area mailboxes for People, Places and Things. Next week we finish up with the remaining six groups. Thanks to all you Area support people for your cooperation on this.
- We have a bunch of unclaimed mailboxes for which we have received no information. We are going to move all these remaining mailboxes to Office 365 and will do the clean-ups and removals from there. There are some 37 special admin accounts that will remain on Exchange for now. So any further Missed Migrations should be taken care of for you, but let us know if there are any stragglers.
- We will host two more Q&A Sessions on Today and Next Wednesday in ITB 2025 from 11:00 AM to Noon and also via Skype:
(some browsers don’t automatically correct the URL, so add https:// )
Dial-in via phone (509-335-2277); enter the following Conference ID when prompted: 595800
As always, report issues to CougTech (email@example.com or 509-335-4357)
With the completion of migrations today, we are now 87.5% complete with Phase 2 of the project and are on track to complete this phase by Thursday June 30, 2017.
At then end of next week, we will review any mailboxes left on the on-premise Exchange server and either delete them or migrate them as appropriate.
In July, we will start Phase 3 which will be focused on beginning the move of all other wsu.edu mail systems to Office 365. Each of these will have to be done separately to ensure the appropriate business needs are maintained during the conversions.
Concurrently, we will start Phase 4 of the project, to migrate our group management tool to Grouper. Once Phase 4 is complete in late Fall 2017, we will have the ability to de-license Office 365 users automatically as they leave WSU or move to other roles.
Mail Migrations are continuing on schedule with completion of Presidential/Provost Units and Tri-Cities so far this week. As more people migrate to Office 365, we are seeing more reports on mail bouncing when sending to other WSU addresses (even others in the same area who have also migrated). Remember that Outlook Clients cache Address Information, and retain some hidden details on the old Mailbox from the on-premise Exchange. If the SENDER clears the Auto-Complete list, either for all cached addresses or for particular ones, that should help alleviate the problems.
Please follow the instructions on the its.wsu.edu web site under Outlook and look at the FAQ’s and scroll down to the Sending section:
When I email a specific person it comes back as a failed delivery. I have verified the email address is correct. How can I fix this?
Why am I getting many failed deliveries when sending out emails?
As always, report issues to CougTech (firstname.lastname@example.org or 509-335-4357)
The migration continues this week with the Presidential Units tomorrow, Tri-Cities on Wednesday, and International Programs on Thursday. The migration is on schedule!
With the change implementation last week, we have resolved almost all of the outstanding items that have caused issues with inadvertent account creation of mail forwarding changes.
IAM is down to one bug to fix:
- Some people are not given an enable RemoteMailBox on premise when their mailbox is created in Office 365. This creates a problem with auto discovery and other things. An interim script is in place to correct this whenever FIM runs (Eg hourly). The correct provisioning solution will not be ready until the end of next week.
Of course, other work is in progress that is Office 365 related, but this is new or routine maintenance
- Help Desk tool change to create Managed Accounts directly in Office 365, including using Shared Mailboxes instead of Departmental Mailboxes.
- Script is being written to handle the licensing of departmental mailboxes programmatically instead of manually as is being done now.
- Script to help with automating licensing the special “pay for” services in Office 365 (Eg Visio, Power BI Pro, etc)
- Upgrade AADC
With this week’s migration of the Voiland College of Engineering and Architecture and the College of Arts and Sciences Mailboxes to Office 365, we have passed the 75% mark in our migration efforts. We only have three more weeks to go for this Phase of the project.
We will continue to host Q&A Sessions on Wednesdays between now and end of June locally in ITB 2025 from 11:00 AM to Noon and via Skype:
Dial-in via phone (509-335-2277); enter the following Conference ID when prompted: 595800
We are still working on resolving the E-Mail forwarding issues that we have encountered in the past few weeks. There were various issues with mail forwarding to outside mail services. To alleviate current and future issues with mail forwarding, we will be making a change to how mail is routed and forwarded this coming Thursday Evening. June 8th.
The change ONLY affects those who have WSU Office 365 Mailboxes who also had their @wsu.edu mail forwarding set to an outside mail service of some type (not the WSU central Exchange service or their WSU Office 365 Mail).
People who are affected will not need to take any action; the forwarding will be set for them automatically. However, after Thursday Evening, they may wish to check their Office 365 Mail Forwarding settings to make sure the change was made correctly. We will be sending those affected a note on how to check their Office 365 forwarding, or you can review the FAQ at https://its.wsu.edu/o365-outlook/
The changes we are implementing Thursday Evening will also clear up some issues with alternate E-Mail aliases, in particular some alumni who had an alternate alias.
Going forward (and we are on track to finish the Office 365 Mailbox Migration by the end of June), all outside mail for people with WSU Office 365 Mailboxes will be set to forward to their Office 365 mailbox. People with WSU Office 365 Mailboxes will need to use the above method in Office 365 to set their forwarding to an outside mail service.
The migration for CAHNRS is complete. At this point, we are 65% complete with the on premise Exchange mail migration to Office 365!
Next week, we will migrate Voiland College of Engineering and Architecture and Carson College of Business on premise Exchange users.
As many new users of Office 365 have probably noticed, when you sign in, many application tiles may be shown that are not actually available for use. To utilize these functions requires the establishment of Office 365 Groups which is a significant design issue and is not part of this current project. These applications may become available at a future date. We will be implementing code this weekend to remove the remainder of these non-usable tiles to eliminate confusion on what services are available for use.
We are migrating CAHNRS over the four workdays this week in groups. Currently we have successfully migrated groups 1-3, with 4 migrating Thursday, and group 5 migrating Friday.
Extra Mailboxes Created Incident
Here is an update for the incident on 5/1 (and some other dates) where an Office 365 mailbox was created for people who should not have one. Or doesn’t need one.
We’ve narrowed the list down to about 1,200 people who we need to address and send them an email that tells them:
- you’ve got this mailbox you may not know about
- you should log onto the mailbox and see what mail is in there (and give them the instructions)
- we have redirected all new mail to go to your NID@wsu.edu forwarding. We are inserting a forward in Office 365 to go to the ADLDS entry they have on record. All these people have an email forwarding away from our on premises exchange and Office 365 mail. Could be gmail, yahoo or even our unix mail, VetMed, Tricities, etc, etc
We have a few more things to verify before sending out this mass email to the 1,200. We are expecting to do this mailing today.
Email Forwarding incorrectly changed Incident
In regards to resolving the incident where on 4/30 thousands of people had their email forwarding incorrectly redirected to Office 365. We are ready to turn the corrected program back on. The count of people impacted is still being calculated. What we will tell them is:
- In order to correct our email delivery as part of the Office 365 project we will insert the forwarding for them in their Office 365 mailbox
- We will be changing their NID@wsu.edu forwarding to to go Office 365. And since we have inserted a forward there, it will be correctly delivered to: and then display their ADLDS entry to their gmail, yahoo or other account.
We are expecting to do this Thursday. When this part is fixed for ADLDS, then new employees with a new Office 365 mailbox will finally get their email populated to correctly go to Office 365. And new student mailboxes will get redirected to Office 365 instead of their personal email.
We continued with our migrations from the On-Premise Exchange to Office 365 Mail, having now migrated both College of Education and Murrow College of communication this week. As of today we have reached 44% of our migration goal, by the end of next week we will be at 64%. We are on track to complete the process by end of June.
We will not be performing any migrations over the next few days due to the Memorial Day holiday coming up, but will be migrating around 2500 mailboxes next week as we move CAHNRS.
There continue to be a very few issues with email forwarding due to the complexities of our systems, but we are actively working to resolve those as they come up. Overall, all units are reporting very positive experiences with the migration process.
To further explain the Office 365 security group setup process, we will be deploying additional documentation for systems personnel today on our website.
Phase 2 is on schedule.
Murrow College of Communications migrates tomorrow and the College of Education migrates on Wednesday. This is a smaller migration week in preparation for the Memorial Day holiday. Next week, we will migrate the approximately 2500 mailboxes in CAHNRS.
We are actively working towards the resolution of an ongoing email forwarding issue. In the past, new students have automatically had their O365 mailbox entered as the forward for their WSU account. They can then re-forward their email as desired from within O365. Faculty and Staff have continued to have the ability to change their email forwarding locally. However, with the move of Faculty and Staff to O365, we have had to stop this nightly forwarding setup process so as not to adversely affect mailboxes during the O365 move. This stoppage means that the correct forwarding process is also turned off for new students. To resolve this, we need to change this business logic, and change the way Faculty and Staff with O365 mailboxes can forward their email if desired. The end result is anticipated to be that anyone with an O365 mailbox will have their mail delivered to their O365 mailbox, and if they wish to forward it, they will need to implement forwarding from within O365.
We completed the migration of AOI/AMS and VetMed (those few that have mailboxes on the Central Exchange) with the exception of 2 (non-people) mailboxes which are being re-run.
We are now approximately 1/3 complete with the migration of on premise mailboxes to Office 365 and are on schedule for completion by the end of June.
10:29AM – Both mailboxes are in O365. This completes the cutover for VetMed and AOI/AMS.
The cut over of Student Affairs to O365 was done last night and this morning. There were a few minor issues with errors that were resolved, except for 5 mailboxes (of the 1100 total) that did not convert. Those are being rerun and will migrate later this morning.
It was discovered that some Courtesy Appointments were no longer being displayed in the GAL. We have developed a resolution to this that was tested in our development environment and will be put into production today to fix this issue.
EAB users have been migrated to Office 365. Tomorrow, Student Affairs will migrate.
We are also starting the review process for Phase 3 (migrating other mail system users to Office 365). This will start with Tri-Cities and the current on premise Unix mail users in July.
The previous issue with migrated Office 365 mailboxes not able to schedule a room that is still on-premise has been resolved.
Also the issue with properly syncing members into the new Office 365 security groups has also been resolved.
We are 25% done with migrations, and will continue next week.
We have completed a large number of migrations in the last few weeks; with today’s conversion of WSU Vancouver and the Grad School we now have migrated 25% of the mailbox we need to move. Note that as this covers WSU Spokane, Everett and Vancouver, most of the mailboxes migrated were not on the Pullman campus, but we have also migrated ITS, some Early Adopters, HRS and Athletics.
Next week we plan to migrate the EAB Advisors on Monday, Student Affairs on Tuesday, then AOI and Vet Med people who have central Exchange mailboxes on Wednesday. The following week we have Murrow College of Communications on Tuesday the 23rd and College of Education on Wednesday the 24th. When we complete those we will have migrated 44% of the mailboxes we need to migrate. The following week (after Memorial Day) we will be moving all of CAHNRS and Extension from Tuesday through Friday. At that point we will be at the 64% mark. June may be anticlimactic.
The EAB Advisors will be moving en-masse on Monday. They are spread all around WSU, so your may be called upon to assist them. They have to move as a group due to the EAB-SSC Calendar system they use. Attached is the list of EAB People. Their EAB-SSC Calendar will be changed to point to their Office 365 mailbox.
The moves that have completed have experienced some issues, but most are fixed by jiggling the handle. Here’s what you may expect:
- Most Windows users migrate OK, a few have had to re-create their profiles. Some Mac users may have also re-create their account to point to outlook.office365.com. This applies to mobile clients like Mac Mail as well.
- Permissions that were granted for people to see other people’s calendars may have to be bumped or re-established.
- We are working to correct an issue with Departmental Mailboxes – currently, these cannot be accessed via Outlook Web Access, but they can be added to an Outlook Client as a new account. This will be rectified and we’ll announce the fix.
- We have an issue with migrated Office 365 mailboxes not able to schedule a room that is still on-premise. We are working on the issue – no known workaround at this time, other than asking someone who still has an on-premise mailbox to reserve the room…
We hope we have re-resolved the E-Mail forwarding issues that cropped up again yesterday.
Also, reports from the field indicate that some Help Desk software that used to point to the On-Premise Exchange system for E-Mails (as https://connect.wsu.edu/ews/exchange.aspx) had to be changed to point to: https://outlook.office365.com/EWS/Exchange.asmx (and then logging in with the correct account with the @wsu.edu extension.)
This morning we reported an issue with people being unable to use web access to reach a Departmental Mailbox that had migrated to Office 365 (the workaround was to add the Departmental Mailbox as an account in a full Outlook Client). This has now been resolved; you can access these Departmental Mailboxes using OWA (go to outlook.office365.com and login in with the credentials using the email@example.com form of the username).
We have received very positive reports for all migrations that have happened this week.
8:00 – The HRS and Athletics migrations completed successfully. We will migrate Vancouver and the Graduate School this evening beginning at 6PM.
12:40 – We are still making the necessary changes to get the Departmental mailboxes licensed in Office 365. Once properly licensed, managers of those mailboxes will be able to log directly into them again.
We migrated mailboxes for both the Spokane and Everett campuses. The migration was started last night at 8PM and took approximately 13 hours to complete for a total of almost 2400 mailboxes. Reports are that users are not incurring any significant issues across those campuses as a result of the migration.
As a rule, we will be starting mailbox migrations at 6PM the night before to ensure that all mailboxes are ready for use at the beginning of the business day.
HRS and Athletics will be migrated tonight.
AADC has run through a couple of clean up syncs now so we are much more confident in its current operation. The forwarding problems have been cleared up. We are still dealing with some small, specific issues and some individual issues but do to the overall complexity of our mail system and the coming and going of employees, we expect these will continue throughout the project.
We will be sending out an email to all on premise mailbox owners today on what to expect during the migration. A copy of this will also be sent out to the wsu-sysmgrs list today.
Scheduled migrations for WSU Spokane and WSU Everett will occur tonight and we will continue through the week as scheduled.
The first full sync of AADC finished last night at 10 PM. We are now into the second sync. We are having some limited success with syncing resources and locations as some have properly moved into Office 365 and most have not so they are not visible by those people already migrated. This is because there is not an O365 object representing these resources and locations yet. We anticipate this will get resolved as these items are synced up. Our Microsoft premier support call yesterday determined that the issue was that these objects have not synced fully. We believe this was confirmed when some objects synced yesterday, but we are still investigating. There was also the theory that the object needed to be modified in some manner, before it would sync. We tested this by only modifying some of the Resources and Locations, and those seemed to have synced. If we modify the rest they should sync over and this issue should be resolved.
Some of those who moved Monday (ITS and Early Adopters) had problems with outside mail being delivered. We are pretty certain it’s related to some of those who had a firstname.lastname defined once upon a time. Those should now all be corrected. CougTech has been holding on NID renames, those will now start being processed.
This morning we started the process again to create a mailbox in Office 365. Still expect a day for it to get through the system.
One drawback is that we are not yet forcing email forwarding for new students who get a mailbox. Historically when a new mailbox is created we force the NID@wsu.edu to be forwarded to NID@email.wsu.edu. This is not happening yet. Instead, they will have their NID@wsu.edu forwarded to their personal email account (gmail, yahoo, etc) that they had registered in myWSU when they first created their NID. This will be fixed as soon as possible. If somebody logs a ticket on this issue, CougTech can change it to NID@email.wsu.edu.
There appear to still be some alumni whose mail sent to NID@wsu.edu bounces because they had an alias set such as firstname.lastname which differs from their NID. No ETA on getting these fixed. Until then, CougTech can update their forwarding to go to NID@email.wsu.edu which will work for them or firstname.lastname@example.org will continue to work.
CougTech is continuing to work through the 175 ticket backlog, with the influx of new tickets significantly slowing.
AADC is still completing it’s first full sync since it stopped last Saturday. After it completes, it will start again, catching up more each time. Microsoft has recommended that AADC be allowed to complete two full syncs before we can expect all sync issues to be resolved. With this in mind, we have rescheduled HRS and Athletics migrations from May 4 to May 10. This should allow for the full two syncs to occur. At that point, we will know if the problems with moving locations and resources to Office 365, and the changing of some migrated user’s forwarding back to their first.lastname is still an issue.
The user migration documentation has been developed and is currently being reviewed within ITS. Since no other areas are moving this week, we have decided to send the reviewed email with the document attached out on Monday to all on premise mailboxes. We will send out the document to sysmgrs earlier as it is available.
We are still manually correcting mail bouncing issues as they are reported.
10:30 AM – Overall the mailbox migration went very smoothly for the vast majority of users. There were some questions that arose on Skype, mobile devices, and on general issues. We are pulling together a document of all of the questions we saw with their resolutions that we will be sending all to all migrating users by tomorrow.
AADC is now running so we can now start creating mailboxes for new students and employees that have been waiting for the past two weeks. However there is a backlog so expect 48 hours to catch up for everyone.
FIM is still paused as we fix mailbox forwards that were inadvertently changed and resolving the business logic code problems. So for today, there have been no new role changes reflected in AD for employees and students. We expect to have that going later today.
There have been several unrelated email forwarding/routing issues in the last 5 days.
From this weekend, there were some alumni whose forwarding no longer worked. This was a result of the Identity System upgrade work in regards to the migration to Office 365. This impacted those alumni who have a NID without the firstname.lastname format. For example, if the person’s NID were email@example.com but once upon a time in Office 365 they were known as firstname.lastname@example.org then those are the ones impacted. These have not all been fixed yet. They will get corrected after several other problems are worked out.
From 6PM Sunday until late last night there were issues with email forwarding for some who had a non-ITS mail server. So mail sent to NID@wsu.edu that was supposed to go to ID@tricity.wsu.edu was bouncing. Most of those have already been corrected. In particular:
Finally, this morning we have had reports of some people not having mail routed properly. This is for people who had a firstname.lastname alternate alias once upon a time. And we’re not certain, but this may have been limited to the ITS folks migrated recently, and the early adopters. Its appearing that this will correct itself in an hour or two as our synchronization with Microsoft completes.
3:30 PM – We have encountered an error with moving resources and locations into Office 365. We are currently engaged with Microsoft attempting to resolve this issue. This resolution is required for us to move forward with area migrations.
We have also completed the document for users so they know how to resolve any potential issues with their new Office 365 account after migration. We will be sending this out to all users with on premise Exchange mailboxes tomorrow.
Happy May Day!
8:00AM – We finally got rid of all the dual mailboxes at 4:00PM last Friday and started the syncing of AADC. On Saturday at 10:00AM, AADC stopped when it ran into a number of items syncing limit. This took a call to Microsoft to resolve, which happened at 6:30PM. AADC syncing was restarted at that time but will now not finish until Monday afternoon sometime. On Sunday at 1:00PM, FIM had to be halted as an issue with X500 addresses was causing an issue in the business logic and FIM was going to create mailboxes for 10,000 former employees. In order to lift the blackout period and ensure that new mailboxes could be created in Office 365, it was decided to just let FIM create the mailboxes and we will delete them later as it has no customer effect. So FIM is running and AADC continues to sync.
Although there are approximately 30 Department/Location/Resource (DLR) mailboxes still to sync for ITS, we have started the ITS and Early Adopter cutover to Office 365. Users will migrate to Office 365 this morning with the exception of these DLR mailboxes which will be synced as soon as AADC is finished. This will allow us to develop the user experience documentation for the migration that will be distributed for future migrations.
12:00PM – ITS and 88 early IT adopters from around WSU have been successfully migrated. We have seen some minor issues but the majority of migrations went smoothly. One noticeable issue was the time between the start and finish of each user’s migration. Some were short (our previous tests showed about a 15 minute impact) and some where as long as 2 1/2 hours. Realizing this is not an acceptable user experience, we have decided to perform the migrations after hours the day before they are scheduled so that users will not be impacted by this delay, but will already be migrated by 8:00 AM on their scheduled day.
- We have migrated ITS personnel and the Early Adopters to Office 365 earlier today. Some issues but generally not too bad (except see below) – please report all problems to CougTech@wsu.edu or 5-HELP as they are compiling FAQs for later migrations.
- We had 155 Dual Mailbox People whose Mailboxes have Merged – the method we outlined last week was superseded by a different method whereby their Office 365 Mail was downloaded and merged into their On-Premise connect.wsu.edu mailbox (expanding the quota where needed). This may not be what some people were expecting, as we had offered a method where we would merge their On-Premise mailbox into the Cloud. In any case, they were notified today to look in their On-Premise mailbox for a folder named something like: ‘email@example.com (Primary)’ and expand that and all their Office 365 Mail, Calendars and Contacts should be in there. When the time comes to migrate their mailbox the whole merge info will get migrated back to Office 365.
- Reminder – we have weekly Q&A sessions in ITB 2025 at 11:00 AM on Wednesday – also via Skype for people to ask questions.meet.wsu.edu/jfdsac/27R7410X
(some browsers don’t automatically correct the URL, so add https:// )
Dial-in via phone (509-335-2277); enter the following Conference ID when prompted: 595800
ITS will take the results of today’s migration experience and prepare a user expectations document that will be sent out to each area before they are scheduled to migrate.
9:00 AM The effort to get the remaining 154 dual mailboxes to export in one batch has failed several times. The problem is that the session is getting interrupted and not resuming, which then requires a new export and new download. Instead of spending more time to troubleshoot this issue we have created 154 new batches and will export them individually. This will take longer, but we will be able to restart exports on individual mailboxes instead of starting the whole batch over again.
10:30 AM The process of creating the individual batches is currently failing at a rate of up to 50%. So we are making headway, but having to reinitialize many of the exports. The large size of some of the mailboxes (up to 7GB), increases the opportunity for communications failures during the export process. We continue to work this process and still have a goal of getting AADC started no later than tomorrow morning.
3:15 PM We are two thirds complete with deleting the 154 O365 dual mailboxes. Once finished, we will be reviewing AADC exports for accuracy. We are still on schedule to restart AADC in production tomorrow and let it sync over the next 3 days, which keeps us on schedule for migrations to begin Monday May 1, 2017.
Our testing has revealed that our intended method for merging the 154 remaining dual mailboxes without AADC in operation will not be successful. So we are now intending to take the mailbox contents and export those to files which will then be imported into each user’s on premise mailbox. Unfortunately, the imported contents will be in a separate folder structure, but all of the contents will be available. Once that is completed, all of the 154 Office 365 mailboxes will be deleted and the user’s affected will have their on premise mailbox migrated to Office 365 with their area.
Once AADC is started (after all dual mailboxes are deleted, so probably tomorrow), there will be a period of 48-60 hours for AADC to be completely in sync. This will require the ITS and Early Adopter migrations to be moved to Monday May 1, 2017 followed quickly by HRS and Athletics on Thursday May 4, 2017.
Although there are still unknowns in the timing required for migrations, a migration schedule has been developed and will be available soon for planning purposes. This schedule is still subject to change as more information is obtained by performing actual mailbox migrations.
ITS and Early adopters now scheduled to be migrated Friday. Will be working on scheduling all areas.
Weekly project area lead meeting status:
- AADC is waiting to be turned back on. Can probably be restarted tomorrow.
- Waiting for all dual mailboxes to be deleted and merged.
- Still looking at some of the export results to ensure AADC will be syncing as expected
- Have tested managed mailboxes DLR with AADC in development. Unfortunately can’t see exactly what happens in production until AADC is turned on.
- When full AADC synced Sunday evening, there were 120 synchronization errors that are being cleaned up. These errors are normally easy to resolve.
- Have 158 accounts to delete and restore as nid.temp mailboxes to get rid of the dual mailboxes for people that needed their mailboxes merged. We will send one time passwords to each of the 158 users to get to their temporary mailboxes.
- 36 other dual mailboxes to be deleted today.
- Need to kick off resource location migration for ITS as soon as AADC is started.
- Migration timing has significantly quickened due to configuration changes that were made locally and on our Microsoft account.
- New email address policies in place for all mail users.
- Updated DLR addresses to ensure validity. Ran into a few issues with addresses that are being corrected.
- Converted over 6000 remote mailboxes.
- Taking the opportunity to clean up the mail environment to reduce risk of future issues with the implementation of AADC.
- Still testing some additional scenarios as we have seen different sync results than expected with AADC due to the proxyaddresses needing to sync back to AD.
- Need to respond back with a couple of test accounts to Jon Walker for EAB testing.
- With Brian Crossland now gone, operational issues are being prioritized as necessary to keep manage both operations and project requirements.
- Kicking off Athletics migration to 95%.
- Some mailboxes have had to be recreated. This is due to users not responding to us and their dual mailbox being deleted with the default process. These are being restored as NID.temp accounts in O365.
- Will start queuing new migration lists for future areas to be migrated as scheduled.
Our goal today is to delete all remaining dual mailboxes, particularly those in CCB and the customers who needed information from both, and requested their mailboxes be merged.
Category 1 – 55
Category 2 – 16
Category 3 – 157
With these lists done, we will have handled over 2838 Dual Mailbox people (plus 913 Single Mailboxes of Separated Employees) for a total so far of 3751 mailbox purged.
We have sent an email to the 157 users in category 3 (wanting their mailboxes merged). Unfortunately, there is not a process that will not inconvenience them for a period. Our Microsoft recommended solution is to move their current O365 mailbox contents to a temporary O365 mailbox not associated with their NID that they will need to access separately with a non-NID password. We will then delete their current O365 mailbox associated with their NID and migrate their on-premise mailbox to O365 along with the members of their associated area. At that time, we can import their temporary O365 mailbox into their newly migrated one and make them whole.
AADC has completed the production sync but we need to delete all dual mailboxes before we turn it back on.
Still looking to migrate ITS later this week if everything stays on schedule.
Continuing dual mailbox deletions:
- 27 Office 365 Mailboxes to purge (Cat 1 Dual Mailboxes)
- 30 Additional On-Premise Exchange Mailboxes to purge (Cat 2 Dual Mailboxes) – one is a Large Number of Items MB (rsvancara)s.
With this morning’s lists done, we will have handled 2225 Dual Mailbox people (plus 913 Single Mailboxes of Separated Employees) for a total so far of 3138 mailbox purged. We are getting there…
This afternoon we will advise the remaining 403 people who have not responded in some way what we are going to do, then will have those lists ready to purge.
On Monday, we will do the 64 CCB people, and 10 stragglers for one reason or another, plus we have 135 Cat 3 people (Merge Mailboxes) – that adds up to 3742 mailboxes accounted for (2838 Dual Mailboxes out of 2881 I have been working with). The remaining 43 Duals are mostly ones we got OOOA messages for so we will convert them to Merges Mailboxes.
We have been testing AADC issues in Development and will convene on Saturday to evaluate our strategy for turning AADC in production back on. Once all dual mailboxes are deleted, and AADC is in production, we can move forward with ITS mailbox migrations to Office 365. We are still looking to do this next week. Once ITS has been successfully migrated, we can start finalizing migration scheduling for the rest of WSU areas.
Continued dual mailbox deletion:
- 913 On-Premise Exchange Mailboxes for Separated (over 4 months) Employees who only had an Exchange MB (Not Dual Mailbox people).
- 236 Office 365 Mailboxes to purge (Cat 1 Dual Mailboxes)
- 32 Additional On-Premise Exchange Mailboxes to purge (Cat 2 Dual Mailboxes)
With this morning’s lists done, we will have handled 2168 Dual Mailbox people (plus 913 Single Mailboxes of Separated Employees) for a total so far of 3081 mailbox purged
Started the deletion of dual mailboxes according to the responses from the mailbox owners.
Category 1 – Delete Office 365 mailbox and keep on premise mailbox for later migration
Category 2 – Delete on premise mailbox and keep Office 365 mailbox, no need for migration
Category 3 – Need mail from both mailboxes, so need to merge into one mailbox (Process still undetermined)
Not Active – Employees who still have mailboxes but have been separated from WSU since December 1, 2016 (Emeritus Faculty exempt)
Cat 1 – 633
Cat 2 – 41
NA – 1226
Although we have directed contacted all dual mailbox accounts (2,881), we have still yet to hear back from almost 800 people. To facilitate the dual mailbox deletions, we have sent out another note to these people and included their IT representative if known. They must respond to us no later than Thursday April 20 or we will perform their dual mailbox based upon their WSU status.
Priorities for the project have changed due to issues with the implementation of AADC, or new data syncing tool making unanticipated changes to existing mailboxes. We have determined our best strategy is to immediately remove all dual mailboxes before starting the mailbox migrations, and we are currently focused on this process. Initially, we had planned to removing the individual dual mailboxes as we migrated those people with their associated areas.
It has been determined that a change to our identity management processes on May 1, 2017 has led to the inadvertent creation of Office 365 mailboxes for up to 2900 employees who did not already have an Office 365 or ITS on premise Exchange mailbox. In this situation, mail sent to one of these users from outside of Office 365 would be delivered to their correct mailbox. However, mail sent by someone within the Office 365 environment would go to the user’s newly created Office 365 mailbox, which the user would not be aware even existed. This would lead to the user not receiving some mail, depending on its source. We now know why this happened and have made changes so that it is no longer happening. The inadvertent change was not noticed at the time as there were very few employees sending mail from within Office 365. However, with 45% of WSU employees now on Office 365, this is no longer the case.
We are currently determining the actual scope of the issue and the specific individuals effected, and then will develop a resolution. The resolution will require changing the forwarding on the Office 365 mailbox to their current mailbox, review of the email in Office 365 mailbox by the user to ensure important emails have not been missed, and then subsequent deletion of the Office 365 mailbox.
Users who are missing mail now and can’t wait for an automated solution, can log into their Office 365 mailbox at http://office365.wsu.edu and sign in with their NID and password. They can then change the forwarding within that account to their regular mailbox and no further accumulation of email in their Office 365 mailbox will occur.
How to Forward your Office 365 Email to Another Account:
- Login to Office 365 (http://office365.wsu.edu)
- Authenticate with your NID and password
- Click the Mail tile.
- Click Settings (gear icon in the upper right-hand of your screen).
- Under Your app settings – Click Mail.
- Click Accounts
- Click Forwarding
- Under the “Forwarding” heading, select Start Forwarding
- Enter the email address you want to forward your email to
- Note: You have the option to keep a copy of all forwarded messages in your Office365 account.
- Select Save.
We are correcting Vet Med personnel first due to the severity of the issue for their users.
It appears we made a mistake last night. We started running the midnight run in FIM for the first time in a couple of weeks and forgot to comment out the ADLDS part of the run. That’s the part that caused our previous inadvertent email forwarding changes.
We are re-processing all the previous changes that we did a week and a half ago to revert these entries to their correct value.
12:40 – We thought we had corrected this issue for all mailboxes at 10:00AM, however EECS mailboxes were missed in the correction. We have now resolved the issues with 14 additional EECS mailboxes.
We had some issues as part of the syncing process that caused the mail forwarding for users who did not have an on premise mailbox to change to point them to Office 365 instead of their correct mail system. It has been reported that this affected some users in Tri-Cities and EECS. This change has resulted in users sending to these addresses having their email bounced back and not delivered.
Alumni reporting emails bouncing
Due to a change in the attributes used for business logic within the implementation of AADC, certain users who had forwarding to @tricities.wsu.edu and @eecs.wsu.edu had their forwarding changed to Office 365 where there was no recipient mailbox. This caused email to bounce that was sent to these addresses. These inadvertent changes have been reversed back to their original forward, and ITS will be changing the associated business logic code so that this attribute is no longer used in this way.
When we updated people on April 13th, we only updated the 27,000 that were broken. But now we have people getting mail for the old AlternateAlias@email.wsu.edu which used to work just fine. But for 2/3 of the people exported Friday and Saturday, when they were exported by AADC the old AlternateAlias@email.wsu.edu entries at Office 365 were deleted. So they don’t work anymore. We’ll write a script to fix everybody in ADLDS email forwarding to use NID@ in their forward instead of AlternateAlias@ in their forward. Please help remind people to clear the Outlook Cache of addresses. For the most part these can be cleaned up by having the WSU Sender clear their Outlook Autocomplete List – in Outlook, File > Options > Mail then scroll down to the Empty Auto-Complete List section and click on the button. This clears the list and new mail should be obtained from the Global Address List.
Several on campus folks, professors and students, who have sent emails as recent as yesterday have said that those emails bounce back to them.
This is happening when a person who had a dual mailbox, had their on-premise mailbox deleted and the person sending the email has the old location cached or is responding to an old email. So when someone tries to reply to an email that was sent before the deletion, they are replying back to a mailbox that no longer exists and the email gets bounced back to them. To resolve this issue, we are to adding an X500 record for these people who had their mailbox deleted so that mail sent to them will be routed correctly to their Office 365 mailbox.
As we have deleted dual mailboxes based upon the user’s response or using our default process, some users have contacted IT saying they need mail that was in the deleted mailbox. To resolve this, we have been restoring the user’s mailbox as a renamed user.temp mailbox.
We are now seeing a new odd behavior occurring with restored mailboxes. Several emails sent to firstname.lastname@example.org have found their way rerouted to email@example.com. The majority of mail to firstname.lastname@example.org have been showing up in the exchange account, but there have been several that have gone to the temp account.
We created the temp accounts and plumbed them to the user’s disconnected Exchange mailbox. They should be isolated unless forwarding of some form has been put in place. The intent for the new temp account is to recover email to the user’s O365 mailbox.
This has been fixed on all of the Temp addresses that were created for on premise Exchange restores, and will be implemented on the O365 restores as well as the O365 restores for the mailboxes needing to be merged.
A few reports of bouncing emails to people who had one of their dual mailboxes deleted, but very sporadic and inconsistent.
Most likely caused by cached addresses on clients to a location that no longer exists. Clearing Outlook cache should resolve. Very limited reports, so waiting to see if this is an issue or maybe just temporary during the deletions.
- Numerous issues occurred last week starting with the attempt to activate AADC in production.
- AADC was turned on at Noon Wednesday April 12th as scheduled. During the next few hours there were reports to CougTech of users not receiving external email. After investigation, it became apparent that there was an unanticipated issue occurring with AADC deleting some proxyaddress values for approximately 27,000 Office 365 accounts including 542 dual mailboxes.
- AADC was turned off around 5:30 PM Wednesday evening to keep it from making further changes.
- The deleted proxyaddress values were restored for the 27,000 accounts at midnight Thursday morning.
- We ended up manually fixing additional individual mailbox forwarding issues throughout Thursday and Friday.
- In addition, on Thursday April 13th, we had multiple reports from EECS of intermittent email delays. Although this was incorrectly attributed to Office 365, it was in fact network spikes that were occurring in the communications between ProofPoint and WSU between 10 AM – 2PM that was causing the delays.
- During all this time since 4/12 the identity management system FIM had been going through the upgrade process to accommodate AADC. On Friday April 14, one of the FIM management agents changed the email forwarding to incorrectly go to the wrong dual mailbox for some 500 customers. Since this was done before actually planned, another 400 non-dual mailbox active customers who had their forwarding route off campus (Eg gmail, Hotmail, etc) were also changed to send their email to their on premise exchange mailbox instead of being forwarded off site.
- On Sunday April 16 at 5:00 PM, we corrected the 900 active employees who had the email redirected to no premise exchange to go to what they had set on April 10th in ADLDS. Note that no email was lost, just redirected to a different mailbox.
- Due to the necessary additions of valid domains, Office 365 and on premise mailboxes are now setup so that dual mailboxes are unable to forward from one to the other (as both systems now contain the same valid proxyaddress names). Users that have dual mailboxes that had email sent to the wrong mailbox will need to maintain both mailboxes to view these emails until the mailboxes are merged/deleted.
- In both of the Office 365 issues above, dual mailboxes have been and continue to be a significant problem different from all the other mailboxes. We originally planned on resolving/merging dual mailboxes in conjunction with the move of each area/department. These incidents showed that dual mailboxes have prevented us from moving forward with the AADC implementation. We are now prioritizing the deletion of all dual mailboxes ahead of the scheduled migrations.
- As mentioned in the AADC production change request, and announced to the WSU community, there was going to be a “black out” period where new student NIDs created were unable to get a mailbox (and other services like ProPlus and OneDrive) in Office 365 for a 5 day period. This is now expected to be a 9-12 day period for new students.
- The first possible time to migrate ITS is now Tuesday April 25th, followed quickly by the Early Adopter Group. Still on schedule to potentially migrate Athletics and HRS May 2-3. If the ITS migration is postponed, we will have to make adjustments to the remaining scheduled migrations.
Microsoft made a change to AADC functionality last December that we were unaware of. We have duplicated all proxyaddresses in both on premise and Office 365 to ensure deletions do not occur in the future.
Not caused by Office 365.
This is the issue that caused external mail to be delivered to the wrong dual mailbox. This has been resolved
The blackout period has kept anyone who got a new NID after April 12, 2017 from being able to get an Office 365 mailbox. We do have the ability to manually implement an on premise mailbox for employees if necessary during this period.