Our Master Service Agreement
Introduction to TCG
Technology Consulting Group is a team of dedicated technology professionals in Stanford's central IT organization called University IT. Our for-fee services to the campus community include systems administration, virtualization, cloudification and solutions engineering, IT consultation and data backup. We are, in some ways, comparable to an internal managed service provider, but we're not-for-profit. TCG uses a cost-recovery model to provide the revenue we need to cover nearly all our expenses. As longtime university employees, we share the goal of advancing Stanford's mission.
Workgroups and teams gain the direct benefit of their technology investments, with access to IT professionals for consultation and guidance—without the expense of hiring a local systems administrator, the day-to-day hassle of performing routine maintenance and operational activities, and without the the worries of data handling compliance.
Master Service Agreement
This is our MSA, which outlines obligations and commitments between our team and our clients. It is separate from any other agreement with University IT.
Our Services
Technology Consulting Group (TCG) provides a handful of different technology services to the campus community. This Master Service Agreement is mostly oriented around our Comprehensive Support agreements. Other facets of our service apply as indicated.
Comprehensive Support for Systems and Servers
Technology Consulting Group (TCG) provides system administration covering nearly every aspect of deploying and managing a server (or group of servers) for the duration of its lifecycle. For a single monthly recurring charge, typically applied to each server that's included in the agreement, almost all supportive activities are covered.
Our Comprehensive Support agreement can cover systems around the clock (24/7) or only during business hours (M-F, 8-5). Either schedule can be applied to development or production servers, but we recommend 24/7 support for all mission-critical systems.
Cloud Consulting and Engineering
We can help your you evaluate your technology needs, manage the risk categories of your data, understand how the cloud can successfully be used at Stanford, be made aware of all options for deployment, look at different Software-as-a-Service products, and successfully tackle large IT systems engineering projects. We use a Time & Materials hourly rate to recover this work.
Endpoint Backup with Code42 CrashPlan
TCG manages central Code42 CrashPlan service, a product that automatically and constantly backs up user data to the cloud, including High Risk and PHI classifications. Today, across campus, there's over 3.5 petabytes of compressed and encrypted data securely stored in the cloud, from over 24,000 devices owned by nearly 13,000 user accounts.
Cloud Backup for Servers
To protect against data loss from earthquakes, floods, fire and fat-fingered accidents, TCG offers standalone managed cloud backup for servers. The same software cloud solution we use for our Comprehensive Support clients is offered as a backup service that can be used on servers we do not manage.
DataDog Systems Monitoring Licenses
DataDog is a cutting-edge cloud-based product that can be used to monitor on-campus servers, virtual environments, applications, databases, cloud resources and user accounts. We use this on the systems we manage, and will resell discounted licenses to the campus.
What's Included
We maintain a list of common activities that are included in our Comprehensive Support agreements. Here are some of the highlights.
Ongoing consultation support for making sound technology decisions
Installation of operating system and third-party software
Virtualization services (including converting from physical systems)
Cloud services, account management, architecture and engineering
Working with your vendors and developers to coordinate a deployment
Backup, restore, and disaster planning management
Control your environment with our change management processes and policies
Support for clustering and load balancing
Virus protection, intrusion detection, firewall management
No extra charge for University IT security-related technologies, like centralized logging (Splunk)
Security scanning and vulnerability remediation
Maintenance, patch management, stabilization, and monitoring
User and workgroup management through tools like Workgroup Manager
Integration with the campus infrastructure
Coordination with other IT support groups
Assistance managing vendors and IT purchases
What's Out of Scope
There are some activities that are not included in our Comprehensive Support agreements. These include:
Custom coding, programming or application development.
Desktop or end-user support, except in some cases where users are attempting to access certain TCG-managed resources. We recommend CRC for this.
Application administration. TCG will install and perform default configuration of most third-party programs and applications, and assist when system-level changes are needed. However, we depend on our clients to run and manage their own applications.
Meeting application-specific minimum security (Minsec) requirements; for example, remediation of SQL injection vulnerabilities in content management systems. We can help configure scanning and vulnerability detection, but the overall responsibility lies with our clients and their developers.
Managing systems that process credit card transactions. PCI compliance is a specialty of our colleagues in Enterprise Technology group.
Deploying high-performance, mainframe, distributed or massive parallel-processing computing solutions. We will gladly introduce you to our colleagues with the Stanford Research Computing Center.
Supplementary Expenses
While our Comprehensive Support agreements cover most expenses relating to the build and operational support of your system, there are some charges that might be required to fully protect and manage your IT resources.
Cloud Accounts
If we build a solution in Amazon Web Services, Microsoft Azure or Google Cloud Platform, we will create a dedicated account where all associated costs within that account are charged against your PTA. This account is for the exclusive use of TCG-managed resources. We'll happily create a second account for non-TCG purposes and integrate that with Stanford's infrastructure.
Backup & Data Protection
Our Comprehensive Support agreement includes management of backups to protect your data. TCG staff will install and configure enterprise backup software to store copies of your files in the cloud, monitor the daily or weekly progress of backups and troubleshoot any issues that might come up. We will also perform test restores and complete restores as needed.
Cloud backup is a cost-effective way to ensure the continuity of your operations should a regional disaster strike or an accidental deletion interrupt your work.
Backup Software Licenses
TCG uses enterprise-grade backup software that’s been approved by Stanford Information Security Office to protect all risk classifications. Depending on your particular system, the software might be configured to encrypt data before it leaves your server and might be stored on encrypted cloud storage volumes. It will always been transmitted over an encrypted network connection, however.
As part of this process, your PTA will be charged a modest recurring monthly charge for the backup software.
Cloud Backup Storage and Restores
In addition to the software licenses to perform backups, our clients’ PTA will be charged each month for the associated cloud storage costs. This variable charge corresponds with the volume of data that’s stored in the cloud.
We select the best cloud vendor and storage tiers based on factors like the expected frequency of performing restores, the speed at which we might need to restore data, the overall volume of data and other client-centric considerations. Every storage provider that TCG uses has already been approved for all risk classifications.
Depending on the volume of data and timing of restores, there will likely be a charge from your cloud provider for data transfer, expedited retrieval, I/O operations and possibly other activities.
Third-Party Support Agreements
Red Hat Agreements
In order to use Red Hat Enterprise Linux (RHEL), a support agreement must be purchased to provide access to patches and software updates. TCG will help with the procurement of the agreement and apply the annual licenses, but any expenses are the responsibility of our clients. We recommend deploying CentOS as a free and open-sourced version of RHEL.
Extended-Support OS Agreements
Operating systems that age into a limited, near-end-of-life or extended-support phase may require a for-fee subscription to receive software patches. The expenses relating to this subscription is the responsibility of our clients.
If the extended-support agreement is determined to be too limited to secure your particular solution, or fails to include the necessary remediation against vulnerabilities in applications or the operating system, TCG will insist on upgrading or rebuilding your server to bring the system into compliance as a condition of continued support.
Other System Support Agreements
Clients are responsible for purchasing and managing necessary support contracts with vendors. TCG does not manage or pay for support agreements with hardware vendors like Dell, or support agreements from cloud vendors like Amazon Web Services, either.
Uncommon Extra Charges
TCG is committed to billing transparency and accuracy. At the earlier stages of your engagement with TCG, we will discuss whether your service requires either an additional complex charge or whether you should expect a time and materials charge in addition to the flat monthly recurring charge. Most systems support agreements do not require either.
Especially Complex Systems
While most of TCG’s services are included in our Comprehensive Support agreements, there may be rare cases where we must apply a supplemental charge to support exceptionally complex systems. We apply this charge to recover extra, ongoing administration overhead. An example of such an uncommon system might be a server arranged as a failover cluster, or a system that requires additional management of attached fibre channel storage.
Research Time & Materials Supplementary Charge
There may be occasions when a given project requires deeper research or a process of discovery, the deployment of a proof of concept or an initial beta system for demonstration and trial use. For these, an hourly time-and-materials charge will be discussed to recover work outside of our Comprehensive Support agreement. Typically, this will be billed at the higher standard UIT rate for “Design & Development” work. Billing at the less expensive “Service Deployment & Administration” would be at the discretion of TCG where appropriate.
Full-Sponsored SUNet IDs
If your collaborators need to log into systems securely and you wish to provide them with a Stanford email address, we recommend a full-sponsored SUNet ID. Our clients are responsible for this modest monthly recurring charge.
Privileged Administrative Workstations (PAWs)
As part of Stanford's Minimum Security requirements, all high risk systems must be administered using a PAW. TCG's technologists have been provisioned a PAW, and regularly use this special laptop to undertake our administrative activities.
If our clients wish to perform administrator activities on their high-risk servers (usually defined as a root or elevated privilege actions), they must procure their own Privileged Administrative Workstations for their constituents. This can be requested using ServiceNow.
Billing for Services
TCG is a service center that uses a cost recovery model; this means we are supported entirely by the revenue we generate from our services. This includes paying for our tools and software, training, staff salary, benefits and impacts our ability to grow our team. The rates and fees we charge our clients recover these expenses. Importantly, our annual budget is designed with the intention of breaking even; we are legally prohibited from profiting beyond 5% of our actual expenses.
When work commences on systems covered under our Comprehensive Support agreements, TCG will apply the associated monthly recurring charge to your PTA to recover the expense of providing this service. This recurring charge will continue each month until the system is decommissioned or the agreement is terminated.
Support Commitment
Our Comprehensive Support model typically covers virtually all support activities relating to your system. To balance out an uneven distribution of work, we expect our clients to commit to at least one year of service.
Support Termination
We hope you find value in our support agreements. However, if that’s not the case and we fall short of your needs or expectations, we want to know as soon as possible for any course correction. We invite you to share your feedback so we can learn and improve. We’ll listen and adjust our practice to make sure you’re satisfied.
However, if for whatever reason, things go awry and we’re unable to keep your trust, faith and satisfaction, we will agree to transition your system’s support to another team and immediately stop billing.
Conversely, if TCG’s client relationship becomes problematic, we commit to seeking a productive, respectful and healthy exchange to provide an opportunity to make improvements. Ultimately, TCG reserves the right to terminate a client’s support agreement if we can't find common ground.
Getting Support
Hours and Days of Support
Normally, our business hours are from 8:00 AM to 5:00 PM, Monday through Friday. We follow the University's staff calendar that informs when we're out of the office for holidays.
Winter Closure
Every December and January, Stanford closes the campus to regular activity for about two weeks between Christmas and New Years' Day. During Winter Closure, we defer regular requests for support until we return in January. Like the rest of University IT, we operate only a skeleton crew during this time, responding only to major outages.
Real-Time Communications
We love chatting with our clients using Stanford Slack Enterprise Grid. This is especially true when setting up a system where we need to work collaboratively. We have shared our #tcg-external channel with the whole Grid, so anyone can search, lurk and join to chat. We're online nearly all the workday (and then some).
TCG typically creates new client-specific channels, just to engage privately with our colleagues. We will share that channel with our clients' workspaces.
Non-urgent Requests and Incidents
Please submit any non-urgent support requests and incidents directly to our team by using our custom support link. This easy-to-remember URL resolves to Stanford ServiceNow, but is hard-wired to our assignment group.
Your ticket will be trackable and viewable by our entire team. If the ticket is best served by another group, we'll route your help ticket accordingly.
We will not participate in client-side support ticketing systems or support mailing lists.
Off-Hours Urgent Support
If you have a 24/7 support agreement, you can contact 24/7 via our support line:
+1 (858) 888-9634
Your call will be immediately routed to the personal cell phone of the on-call TCG team member — so please exercise this option only when there are urgent off-hour situations.
Business Hours Support
If your system is has a business-hours only Comprehensive Support agreement, we will undertake any work at the next available workday.
Response Commitments
We commit to responding expeditiously to alerts, outages and incidents. However, like the rest of University IT, we can't commit to a resolution period.
Non-Urgent Support Requests via ServiceNow: One business day.
Urgent Support Requests via Phone: One hour to return any left voicemails, if not immediately connected.
Incident Alerts: Acknowledged within 90 minutes for 24/7 Comprehensive Support agreements, the next business day for after-hours alerts for Business Hours-only Comprehensive Support agreements.
Escalations
Calling our Off-Hours Urgent Support telephone number (858-888-9634) will give you the option to be connected to a manager or director to escalate an issue.
You can contact the Director of TCG, Noah Abrahamson, using the Stanford-internal directory view of his contact information. He can also be reached via Stanford Slack.
Our Email Address
You can send our whole team an email by sending something to our Office 365 Group address: team@tcg.sh.
Our 7-Step Process
For every Comprehensive Support engagement, we use a custom, established approach to guide our efforts throughout the lifecycle of our clients' solutions, from inspiration to ongoing operational work to ensure uptime and availability. We are a team of consultants that take our responsibility to educate and guide our users, protect Stanford data and advance the mission of the University seriously.
Consult
...through an engagement that helps determine needs, review risk categories, understand cloud options and achieve alignment with department and school initiatives. We will sit down with you, your stakeholders and collaborators to learn about your goals, limitations and priorities. We'll answer any questions, give guidance and connect with additional resources within University IT, as appropriate.
Sketch
...out a technology solution with the right components, integrating the campus infrastructure and services, and provide a back-of-the-envelope estimate. This might include a mix of technologies and possible third-party vendor support options.
Propose
...something more formal, if we decide to move forward. We'll provide a well-reasoned cost estimate, a sound deployment strategy and a plan for long-term operations with ongoing support.
Build
... to specification, setting up servers, virtualizing and deploying as needed using modern practices; integrate billing with Stanford Oracle Financials; directly engage with your stakeholders throughout the process.
Operationalize
...your system, integrate monitoring and schedule routine maintenance, protect data with backups and ensure continuous, responsive availability. This is the day-to-day management stage of your IT resources.
Audit
...your solution periodically to ensure security, compliance and prevent unauthorized access. This measure helps secure Stanford data, protect your resources and identify potential issues for remediation.
Tune
...systems to right-size the technology, optimize performance and taking advantage of any discounts available from our cloud vendors. This limits the risk of over-paying or underutilizing resources and makes the most of your IT funding.
Security
Operating System Versions and Support
TCG supports systems that run modern versions of Red Hat-related distributions of Linux (including CentOS, Oracle Enterprise Linux and Amazon Linux 2), Debian-related Linux (including Ubuntu) and systems running Windows Server operating systems.
End of Life OS
Security compliance at Stanford requires operating systems that are updatable with regular releases of software patches. For this reason, any OS that no longer enjoys mainstream support by the vendor or community of developers cannot be supported by TCG.
Patching and Security Activities
Keeping your operating system and software up-to-date is a necessary and critical component of protecting Stanford’s digital assets. Even servers that exclusively hold fully publically-accessible data are important to secure, because scripted and automated attacks will exploit a vulnerability. Once compromised, cybercriminals are in a beneficial position to transit across our network to compromise other systems.
Update Frequency
Security compliance at Stanford also requires TCG to patch Linux systems no less frequently than every 90 days; Windows systems will be patched no less frequently than once a month. Be advised that these are minimum frequencies. Depending on the results from security-related efforts, actual patching and maintenance frequency might be greater.
Urgent Security-Related Activities
There may be times when high-risk software vulnerabilities are discovered on your system. For this reason, urgent remediative activities may be required. While we will attempt to accommodate our clients’ need for uptime and consistent availability, our necessary emergency response to certain high-risk vulnerabilities might supersede this accommodation and could disrupt your business operations.
Maintenance Windows
For any system with a business-hours only support agreement, regular operational maintenance will be performed exclusively between Monday and Friday, 8:00 AM through 5:00 PM (notwithstanding urgent security-related activities that might occur).
For systems that enjoy 24/7 support, three additional off-hours maintenance windows are available. These windows correspond with the published University IT-wide maintenance windows found at https://uit.stanford.edu/service/changemgt/maintenance-windows. All times are local.
Thursdays 4:00 AM – 6:00 AM
Saturdays 5:00 AM – 8:00 AM
Sundays 5:00 AM – 8:00 AM
Maintenance Freeze
During certain times of the year, University IT pauses most regular activities on systems that have the potential to disrupt important campus activities. Information about maintenance freezes is published here: https://uit.stanford.edu/service/changemgt/freeze; example periods include:
Commencements
Close of the Fiscal Year
Close of the calendar year, donation and grant processing
TCG-managed systems that don’t have the potential to disrupt these activities might not be included in this freeze. For example, file or application servers used exclusively by a department or workgroup whose operation doesn’t affect central billing, academic activity are likely not affected by these freeze events.
Work Initiation
TCG will submit a ServiceNow Change Request prior to undertaking any potentially disruptive work.
Clients will identify the internal contact who will receive requests (by email) to approve or deny the request to commence work. An expeditious response is important. Without approval, work cannot be undertaken and may require rescheduling.
Our clients' business and/or technical contacts have the responsibility to communicate outages or disruptions to a workgroup or stakeholders.
MinSec Modifications
We recognize security-related software, especially when configured to support the highest levels of data risk classification, can be frustrating to encounter. It may impede, thwart or delay normal work performed by our clients. For example, application binary whitelisting will prevent any unauthorized software modifications, while Privileged Access Workstations can be expensive.
Our clients, like everyone at Stanford, are expected to respect these data handling requirements listed in Chapter 6 of the Stanford Admin Guide and the associated Information Security Office and Privacy Office pages.
Clients must not tamper with, configure or remove software managed by TCG for these purposes.
We will work diligently to minimize the impact of any required software on the performance of our client systems.
Other Management Software
TCG uses enterprise systems management tools to configure and administer systems at scale. We use automation when possible to minimize error. For this reason, we require software like BigFix, Puppet and Site 24x7 to be installed and configured to our specifications.
When inheriting a system supported previously by another team, we will evaluate whether we require the server to be rebuilt based on our standard configuration. At our discretion, we may limit our modifications only to installing and (re-)configuring management software to our specifications.
Client Contacts
We ask every new client to provide at least one technical and one business contact. It can be the same person, but not a mailing address or group IMAP account.
This information is used for notifications for events like patching, change request approvals, our client mailing list, billing questions and other business-related purposes.
We ask that our clients keep us informed of any internal organization changes, new PTA billing accounts and other events that might affect the delivery of our service.