Go‎ > ‎

Tech Systems Status Log

BB&N Network Services Incident Log:
 Current Issues:
      • All System Go.
You can also check the Google Servers Status Page and the Haiku Status Page.

  • 11/19/18: 8:15am Internet service at 46 Belmont reported as down; secondary internet activated manually at 8:40am.
  • 10/27/18: SeniorSystems hosted service down for planned maintenance all day window.
  • 10/10/18: Microsoft Office download site stopped working. They provided us a separate web URL as a work-around.
  • 10/6, 10/12, 10/20/18 PowerSchool Learning conducted system upgrades which resulted in system downtime on these three dates.
  • 9/25/18 4:30pm Planned maintenance to upgrade WAN bandwidth will cause minor interruptions. Concluded at 6:30pm.
  • 9/13/18 8:00am WiFi down at much of LS and MS Science/Carriage due to a wireless controller upgrade issue affecting switch configs. Fixed at 11:00am
  • 9/12/18 5:00pm WiFi down due to unplanned maintenance to upgrade wireless controller went unexpectedly long. Resolved at 9:00pm

  • 6/13/18 10:00am US & LS internet interruptions caused by a misconfiguration of our firewall. Fixed at 10:30am.
  • 4/4/18 11:00am Intermittent internet issues at MS and 46; we think due to problems with Comcast. Issues resolved themselves.
  • 3/1/18 2:00pm Significant network outage caused by power loss to servers due to IT error. Servers were brought back up at 4:30pm.
  • 2/16/18 7:00pm Planned maintenance to our core switch resulted in a longer maintenance window than expected. Full service restored at 12:00am.
  • 1/26/18 8:00am Several servers down (and wifi for some clients) due to a hardware failure on a primary server. Rebooting the server solved the issue at 12:30pm.
  • 12/14/17 6:15pm Minor intermittent connection delays on our primary internet connection at US. Some VPN connections drop. Resolved at 8:00pm. During an upgrade, one firewall lost its HA configuration.
  • 12/1/17 4:00pm Internet slowdown at US/LS caused by firewall failover testing. Resolved at 5:00pm
  • 11/7/17 3:00pm Due to a firewall update we had several brief periods of internet downtime at US and LS caused by an old route on the core switch. Issue was resolved at 11pm.
  • 10/4/17 6:00pm PowerSchool was down. Resolved by vendor at 11:00pm (This is a hosted service not under our control.)
  • 9/14/17 8:00am: Student wifi network "BBN WiFi" was not working. 10:00am: Resolution: a DNS issue with our internet provider was causing the problem. We added secondary DNS servers to this network.

  • 2/28/17 9am: Haiku PowerSchool is experiencing an issue of no images displaying and no files available. Resolved at 1pm. Problem found to be an issue with amazon web services as explained here.
  • 1//24/17 11am Primary internet service provider is down. This affects US, LS, and 46 because all three of these campuses are using the back-up connection making it slow for all. Level3 has reported they also see the problem and have escalated the issue to the local carrier. We are waiting for an update from the local carrier. Issue was resolved at 4:00pm.
  • 12/8/16 1:30pm: Phone system outage caused by Verizon work in our server room. Resolved at 3:00pm by powering LightSpan equipment back up.
  • 12/16  MS internet interruptions occurred multiple days for a few minutes each day after lunch (E Block). Resolved after several weeks when Comcast fixed their cable and we added second modem and we blocked student access to entertainment streaming media.
  • 10/17/16 2pm Internet down at MS. Resolved after about 90 minutes. Problem appeared to be with Comcast cable modem.
  • 9/20/6 11am Internet down at 46B due to problem with Comcast. Auto fail-over did not work. Problem resolved around noon. Later found file-server also down; rebooted, resolved. 3pm.
  • 9/1/16 7am Wireless problems at LS, MS, and 46B caused by an upgrade to our system. The wired network is working fine. 12:00pm majority of wireless service restored. 8:00pm back to normal operation. 
  • 8/31/16 6pm  Wireless network down as we do planned replacement of controllers. 8pm update: the new controllers are in place and things are back to normal at US.
  • 8/25/16 - 8:00am Internet access at 46 Belmont has been disrupted. 8:30am update: Internet is back to normal. We believe this outage was due to a problem on the provider's end.
  • 8/22/16 - At 3:00pm we initiated a planned upgrade of software on our core switch. All network services (affecting all campuses) were brought down in order to do this, including phones, wifi, internet, and servers. Unfortunately, the newer software on the device was incompatible with three components inside the device. We have initiated a 4 hour replacement shipping of those components from the provider. At this time the only campuses with internet access are 46 Belmont and Middle School.  8/23/16 - 9:00am Update: This morning we received and replaced the network components and are working with tech support to bring everything back up. The student WiFi network (with internet access) has been working this morning at 46 Belmont, Lower School, and Middle School. Phones are working. 2:00pm update. All network services fully restored following successful configuration of 3 new switch modules. (wireless, supervisor, 48 port blade).

  • 8/1/16 - Internet access was disrupted for LS, MS, & US for half a day due to the expiration of a license file on our firewall. The license had been renewed the previous week, but the new license number had not been applied. Service was restored by applying the new license key to the server.
  • 5/12/16 - 11:55am: Internet running on backup connection for 2.5 hours. Problem determined to be at US internet provider (Level3). Problem resolved and back to normal at 2:45pm 5/12/16.
  • 10/26/15 - 3pm: Internet and phones were down at 80 Coolidge for approximately 15 minutes. Core switch reported temperature problems that resulted in the switch rebooting. We are upgrading the iOS.

  • 5/5/15 - 12pm: Haiku is having trouble with its servers. You can view Haiku server status at this URL: http://status.haikulearning.com/. This link is always available here on our server status page. This issue was fixed by Haiku around 3pm.
  • 4/28/15 - We are experiencing intermittent network problems at our new office location at 80 Coolidge Hill Rd. Investigation and remediation is underway with Comcast. Issue was resolved 4/29/15 by replacing the cable modem.
  • 4/12/15 - 6pm: Major internet and phone outage at US caused by 6509 module failure. Replaced part at 5am on 4/13/15 and all systems are now normal.
  • 2/12/15 - 7am: We are investigating reports that the BBN Faculty Wifi is not working. BBN Student WiFi is reported to be OK.  7:15am Problem resolved by fixing the wireless controller's connection to Active Directory.
  • 2/11/15 - 9am: Student WiFi network down at upper school: We are investigating a problem with the Bradford Network Access Control appliance. 9:15am problem resolved.
  • 2/10/15 - 7am: An overnight power failure (as a result of battery backup failure) resulted in the WAN connection appliance failure. The WAN failure results in no internet access, no email, and no phones working. Verizon arrived at 7:00pm and replaced the box which fixed the network issue at 8:00pm.
  • 2/7/15 - Core network switch had a blade failure at about 7pm which resulted in remote access to network being disabled and several servers were unavailable. We initiated an overnight RMA on the failed module with Cisco and replaced the it at 9am on Sunday. Additionally, we changed the connections on affected servers to connect them to two separate blades so that they have redundancy not dependent on the same blade.
  • 11/19/14 - 6:15am: BBN Faculty WiFi down at US. Resolved at 7:00am. (WCS connection to LDAP had a problem.)
  • 9/15/14 - 10am: Our internet connection is having occasional slow spots. We have a ticket open with the internet provider and are investigating. Resolved.
  • 9/12/14 - Our primary internet provider has bounced down a few times today and possibly yesterday. When this happens our internet fails over to a backup connection but causes momentary downtime. This caused momentary "failed to send email" message. Resolved. 
  • 9/9/14 - FileMaker server software failure overnight. Brought back online at 9:30am.
  • 8/30/14 - 10am: "BBN Faculty" wifi network non-joinable, caused by system clock setting. Resolved at 5pm.
  • 8/28/14 - 1pm: Firewall testing results in some incorrectly blocked sites. Resolved at 3pm.

  • 6/11/14 - 8am: investigating issues with WiFi on multiple campuses. Resolved at 9am.
  • 5/19/14 - 9:00am Wireless network access (WiFi) is down at Lower School. 10am We are engaging Cisco support to troubleshoot. 11:30am: resolved.
  • 4/17/14 - 8:00am: We are troubleshooting a problem with our FileMaker server. It is currently inaccessible. 11:30am: resolved.
  • 1/22/14 - Forbes admission house will be offline this evening due to a switch failure.
  • 11/8/13 - Haiku users may have experienced down-time due to a DDoS attack on Haiku
  • 10/21/13 - 10/22/13:  Verizon internet outage resulted in VPN failure and use of BB&N backup internet connectivity.
  • 9/23/13 - Google Gmail experienced some sending/receiving delays. They are working on resolving it.
  • 9/10/13 - an issue at ZoneEdit with our DNS caused some messages to bounce back as undeliverable between 2 and 6pm.
  • 8/23/13 - FileMaker Pro server was inaccessible for the morning due to a failed windows server update.

  • 7/29/13 - Our service that takes a copy of voicemail and sends it to email was down. All voicemail is accessible through normal channels. Service was restored after updating our smtp server.
  • 01/11/13 - Some users experienced problems over the past 24 hours with the "BBN Faculty" wireless network caused by a mis-configuration as a result of a recent update. Issue has been resolved.
  • 12/17/12 - Network issues at LS this morning have been resolved.
  • 11/26/12 - 11/28 - Faculty wifi network outages, affecting all campuses. Address pool for BBN wifi exceeded. New "BBN Faculty" SSID created and configured for use. Systems back to normal.

Subpages (1): Network Maintenance Log