ITS Alerts

Syndicate content
The Information Technology Services (ITS) Alerts System is a rapid-delivery method used to distribute timely information on major ITS services, unexpected outages, and maintenance schedules to the Penn State community.
Updated: 16 min 40 sec ago

Resolved: [Outage Report] 11/6 Data Center Firewall (DCFW) Service Outage

Wed, 11/12/2014 - 10:45
The following is information about a networking outage that occurred on November 6, 2014.

Overview: At 11:14 p.m. on November 6, several server clusters lost network connectivity for varying amounts of time. The outage was caused in part by an ongoing network infrastructure project (see Alert 3218: http://alerts.its.psu.edu/alert-3218) to relocate Penn State core router and fiber facilities in the Computer Building into a new space.

Details and Cause: On October 28, one of two fibers connecting two central switches was cut as part of the network infrastructure relocation in the Computer Building. Because of the redundancy in place, the broken link went unnoticed until the second fiber connecting the two switches was cut as part of the move around 11:00 p.m. on November 6.

When the second link was cut on November 6, the two switches were separated and the split-brain set up of the master-slave configuration kicked in, disabling all of the ports on the secondary switch. This split triggered a variety of network outages for different clusters, ranging from outages that lasted only a few minutes to extended outages due to additional cut fibers.

It was when the second link was cut and the switches disconnected that the original faulty link was discovered. This discovery revealed that the proper monitoring that would have detected the cut fiber was somehow deactivated before October 28.

Resolution: At approximately 4:00 a.m. on Friday, November 7, the first link was replaced, reconnecting the two switches. At 4:30 a.m. the outage was resolved when the second link was reestablished, producing full connectivity and redundancy. Monitoring was also restored for the equipment at this time.

Future Prevention: In order to prevent outages such as this from occurring in the future, Information Technology Services (ITS) is working collaboratively across subunits to develop and implement improved redundancy testing plans and processes. Starting this week, ITS will also be holding regular topology reviews of different complex clusters so that root causes of outages can be identified more quickly in the future. IT staff will continue to be vigilant about monitoring to avoid lapses such as the one that occurred in this situation.

Thank you for your understanding and patience as work continues on the network infrastructure relocation to provide the Penn State community with expanded and improved networking service.

Resolved: ITS Streaming service maintenance

Wed, 11/12/2014 - 07:43
The ITS Streaming services will be unavailable during the 5am to 7am maintenance window on November 11, 2014. During this time, technical staff will be performing server maintenance on the servers that provide this service. We expect the service to be fully available by 7am.

Resolved: Sites Maintenance

Wed, 11/12/2014 - 06:11
Maintenance has been completed. Staff will continue to monitor the service throughout the day to ensure it\'s integrity. Any questions or concerns please email: sites@psu.edu

Resolved: Production AIS Service Degradation - November 10

Tue, 11/11/2014 - 08:06
On Monday, November 10, 2014, at 10:59pm the following production Administrative Information Services (AIS) services may experience a short outage (lasting up to 10 seconds) while the AIS Networking Team switches over to redundant systems due to a large network infrastructure relocation project (as described in ITS Alert #3218, http://alerts.its.psu.edu/alert-3218):

ais-vpn.ais.psu.edu
AIS website (ais.its.psu.edu)
ANGEL
College Cost Estimator
Course Substitution Request System (CSRS)
Curriculum Submission and Consultation System (curriculum.psu.edu)
Data Warehouse
eCommerce
Edds/Pedds
EIS/Cognos
eLion
eSteward
eStore
Hats
Imaging/Docfinity
iTwo/OBIEE
LionPath
Modulo
Penn State Administrative Web Suite (PAWS)
Student Rating of Teacher Effectiveness (SRTE)

Other services may be impacted as well.

Resolved: Library interruption - A - Z list, November 10, 2014

Mon, 11/10/2014 - 18:09
Services have been restored.

Resolved: Resolved: Turnitin Security Patch Installed

Mon, 11/10/2014 - 17:41
Turnitin has installed a patch around 4PM EDT on Mon., Nov 10 that resolves an ongoing issue where some instructors in Turnitin were able to access another person's profile in Turnitin. The issue was sporadic, but should now be fully resolved. Anyone who does see someone else's profile should report it immediately to turnitin@psu.edu.

WebMail Web Server Upgrade - November 12

Mon, 11/10/2014 - 17:03
On Wednesday, November 12, 2014, during the regularly scheduled maintenance window (5:00-7:00 a.m.), Information Technology Services (ITS) will upgrade the WebMail Web Server.

Resolved: Resolved: ITS Streaming Video Service Outage

Mon, 11/10/2014 - 16:07
On Friday November 7th at 9:50pm, the ITS Streaming Service became unavailable and any media content was inaccessible. The service functionally was restored on Sunday November 9th at 8:21 am. At this time, the service\'s help documentation and FAQs are unavailable. We are working to get this documentation restored to our website as soon as possible. We apologize for any problems this may have caused. If you have any questions, please contact us at streaming@psu.edu

Service interruption, ScholarSphere, 11/12/2014

Mon, 11/10/2014 - 10:48
ScholarSphere will be unavailable during the 5-7 a.m. (EST) maintenance window on November 12, 2014 in order to apply updates.

Resolved: Scheduled Power Shutdown at Hosler building on Nov. 5 & 6, 2014

Fri, 11/07/2014 - 07:08
OPP has scheduled an electrical services shutdown at the Hosler building from 10:30 pm Wednesday, November 5th, to November 6th, 2:30 am.

CLC labs and services in the Hosler building may not be available during this time.

Resolved: VM Hosting Unavailable

Fri, 11/07/2014 - 04:17
As of 3:08am, VMhosting services have been restored. VMhosts are functioning normally and are available.

Resolved: Scheduled Power Shutdown at Willard Building on Nov. 6, 2014

Thu, 11/06/2014 - 14:13
OPP has scheduled an electrical services shutdown at the Willard building on Thursday, November 6th, from 4:30am to 6:30am.

The CLC labs in the Willard building may not be available during this time.

Resolved: UCS Add on Update

Thu, 11/06/2014 - 08:54
Following to this morning's UCS upgrade, one of the UCS zimlets needs to be reinstalled on one of the backend servers. This was first noticed at 7am. Work will take place at 7:50am.

Resolved: Sites Maintenance

Tue, 11/04/2014 - 17:33
This maintenance has been postponed.

Resolved: UCS Outage

Tue, 11/04/2014 - 13:07
There have been no reports of users receiving a network service error since Monday afternoon.

Resolved: Production AIS IdS Service Degradation - November 3

Tue, 11/04/2014 - 13:04
On Monday, November 3, 2014, at 10:59pm the following production Administrative Information Services (AIS) and Identity Services (IdS) services may experience a short outage (lasting up to 2 seconds) while the AIS Networking Team switches over to redundant systems due to a large network infrastructure relocation project (as described in ITS Alert #3218, http://alerts.its.psu.edu/alert-3218):

chat.psu.edu
clubs.psu.edu
Core E-mail (email.psu.edu, authsmtp.psu.edu, and smtp.psu.edu)
Course Online Account (COLA)
Course Substitution Request System (CSRS)
Departmental Web Space
downloads.its.psu.edu
Kerberos
LDAP
Listserv
Name Service
Penn State Access Account Dashboard (www.work.psu.edu)
Penn State Access Account Storage Space (PASS) - Those clients maintaining persistent mounts of PASS may require a remount operation after 10:59pm on November 3 to fully restore functionality
Penn State Homepage (www.psu.edu)
Penn State News (news.psu.edu)
Penn State WebMail
Portal (portal.psu.edu)
protected.personal.psu.edu
RSK6 Lab
SFTP.personal.psu.edu
Tivoli Storage Manager (TSM)
TrackITS (trackits.psu.edu)
University Collaboration Suite (UCS)
Virtual Domain E-mail Hosting
VPN.its.psu.edu
WebAccess
Windows Software Update Service (WSUS)
www.personal.psu.edu

Other services, not listed in this alert, may be impacted as well.

Scheduled Power Shutdown at Willard Building on Nov. 6, 2014

Tue, 11/04/2014 - 11:20
OPP has scheduled an electrical services shutdown at the Willard building on Monday, November 6th, from 4:30am to 6:30am.

The CLC labs in the Willard building may not be available during this time.

Resolved: Scheduled Power Shutdown at Willard building on Nov. 3, 2014

Tue, 11/04/2014 - 10:13
OPP has scheduled an electrical services shutdown at the Willard building on Monday, November 3rd, from 4:30am to 6:30am.

The CLC labs in the Willard building may not be available during this time.

Resolved: ANGEL archive scheduled for October 9

Thu, 10/30/2014 - 11:44
On Wednesday, October 9, 2013 at 4:00 a.m. Eastern Time, Information Technology Services will archive:

* All ANGEL courses from summer 2012 and prior

* All ANGEL groups with low activity for six months or longer


For details, please reference the ANGEL Archiving Policy at http://kb.its.psu.edu/cms/article/71#_ANGEL_policy



Please contact ANGEL Support with questions: angelsupport@psu.edu

Update: UCS Upgrade to 8.0.8

Thu, 10/30/2014 - 10:18
This upgrade will now take place on Thursday, November 6, 2014