March 5 Day 1 Report

posted Mar 5, 2012, 8:15 PM by Rita Girardi   [ updated Mar 7, 2012, 5:59 AM ]
We will provide daily updates after each major phase of the project until technical issues and support issues return to a moderate level.  The March 5 Go-Live occurred today.  

Today's Go-Live
  • Students, Retirees, U-M Online subscribers, and December 2011 graduates were provisioned with all of the Google tools and provided with access to the Migration Assistant to move their IMAP email and contacts into Google. This communication was sent at 4:30 pm on Monday.
  • Faculty and Staff received access to all of the Google collaboration tools, except for email and calendar. This information was communicated to the Unit Leads to cascade to faculty and staff.
  • Sponsored Affiliates received access to all of the Google collaboration tools, except for email and calendar. The communication to the Sponsored Affiliates will be sent out later this week.
Provisioning and Migration Summary
With 113,293 faculty, staff, and students to be provisioned from MCommunity into Google through the connector tool, provisioning began at 1:45 pm on Wednesday, February 29. Provisioning happened in the following order:
  • Dec 2011 graduates
  • UMOL subscribers
  • Retirees
  • Faculty, staff, temp staff and sponsored affiliates
  • Pilot users (data sync only)
Provisioning and quality checks were completed by 11:00 pm on Sunday, March 4.  A list was generated from MCommunity to load into the Migration Assistant.  The Migration Assistant is the self-serve tool that allows users to select which IMAP data to migrate into the new Google account.  The data was cleaned up and loaded by 4:00 pm on Monday, March 5.

Students began to use the Migration Assistant as soon as the data was loaded.  As of 10:30 pm, 1,806 were in the queue, 700 were being migrated, and 8 were completed. 

Support Summary
The number of tickets submitted to the Transition Desk are included below:

Saturday, March 3
  • Migration questions: 2
  • General usage: 3
Sunday, March 4
  • Migration questions: 11
  • Access problems: 3
  • General usage: 4
  • Conflicting account: 2
Monday, March 5
  • Migration Assistant / Migration questions: 112
  • General usage: 37
  • Access problems / questions: 16
  • Conflicting account: 6
  • Email client: 6
  • Mobile devices: 5
Nine tickets were escalated to Tier 2 and one was migrated to Tier 3.  
    Project Team members and Student Google Guides were on-site at the Hatcher Library on Monday from noon to 3:00 pm.  The main purpose for this week’s User Labs is to provide support to faculty and staff regarding the use of the Google collaboration tools.  Although, students and retirees also stopped by to ask questions.  A total of 14 users were provided assistance.  The breakdown of the support included:
    • Collaboration Tools: 3
    • Email: 14
    • Calendar:  1
    • Mobile devices: 2
    • General migration questions: 5
    Campus Computing Sites, Peer Academic Success Specialists (PASS), and Campus Information Centers were also available to provide assistance.

    • Some pilot participants were moved from the “All Tools” sub-org into the “All Tools Except Email and Calendar” sub-org during the provisioning process.  For most users, this was caused because of a Medical School/University of Michigan Hospitals and Health Centers affiliation since these affiliations are not to have Google email and calendar.
      • Corrective Action.  The primary appointment of these faculty and staff was identified.  For those whose primary appointment was a non-health system related unit, the restricted override was set and the user was moved back to the “All Tools” sub-org.  For those whose primary appointment was a health system related unit, a request was submitted to and approved by the Medical School Unit Readiness Coordinator to have these users moved back to the “All Tools” sub-org.
    • After approximately 24 hours of provisioning, the load on the MCommunity server was impacting the entire MCommunity environment.  
      • Corrective Action:  Increased the number of processors dedicated to this server.  
    • A small percentage of the users (<100) got double-placed. An error record was created in the root container and actual user profile in a different (correct) container.
      • Corrective Action:  This was manually corrected for those affected users.
    • New students from the Office of New Student Programs were getting sent mail even though they do not have a working Google mailbox yet. 
      • Corrective Action:  Manually add mail forwarding addresses to these users on Monday.
    • 1693 users were not provisioned into Google even thought they have a qualifying institutional role.  This happened for these alumni users with multiple roles.  
      • Corrective Action:  The 1693 users were manually provisioned into Google.  A fix to the connector tool will be developed and uploaded.  
    • The mail forwarding address for 89 pilot participants was removed during the provisioning.  This was caused by an incorrect filter.
      • Corrective Action: The mail forwarding address was added for these users.
    • The Transition Desk was fully staffed earlier in the day with constant and manageable traffic.  The Transition Desk experienced an increase in call volume around 5:45 pm causing dropped calls and high hold times.  Call volume settled to a manageable level around 6:15 pm.  The increased call volume was likely a result of students being done with classes and reading the communications that were sent out.
      • Corrective Action: Validate timing of outgoing communications with Transition Desk Supervisor for proper staffing.
    • The Retirees, U-M Online subscribers and December 2011 graduates list differed from the users loaded into the Migration Assistant.  This resulted in 3,256 users receiving communications about the March 5 migration (Note: They were excluded from the March 5 communication).  Also, approximately 10,000 additional users were loaded into the Migration Assistant that did not receive communications.
      • Corrective Action:  Additional analysis is required to evaluate the differences, fix the problem and communicate the solution.  
    • During validation of the Migration Assistant list it was determined that 37 users were not provisioned from MCommunity into the proper sub-org.  This specifically involved users that were both a pre-December 2011 alumni and U-M Online subscribers.
      • Corrective Action: Manually move users into right sub-org, load them into the Migration Assistant and send communications.
    • During validation of the Migration Assistant list, it was determined that 99 users were not provisioned from MCommunity into Google.  This error occurred for new accounts with special privacy settings.
      • Corrective Action: Manually provision users into Google, load users into the Migration Assistant and send communications. A fix to the connector tool will be developed and uploaded.  
    • During validation of the Migration Assistant list, it was determined that 2 users were improperly left in the Team Edition sub-org.
      • Corrective Action: Manually migrate users to correct sub-org and verify with MCommunity of proper placement.
    • IMAP backend servers fell behind in replication during ramp up of self migration Google Apps Migration for Microsoft Exchange (GAMME) servers
      • Corrective Action: Delayed turning on more self migration GAMME servers. Took actions to improve sync performance. Monitored sync times and adjusted incoming load. IMAP backend server performance and stability will improve as more users are transferred.
    • The connection between the Migration Assistant and the GAMME servers is not reporting IMAP contacts processed back to the API. The outcome is that the Migration Assistant reports that the contacts are still processing when they are done.
      • Corrective Action: Short term generate lists of processed contacts and write a script to touch proper API URL.  The long-term solution is to import current working settings environment into Dev with debug mode turned on and troubleshoot further.