Leaders of Industry: When Protecting BMS Assets, What Does ‘Good’ Look Like for Big Ten university PSU?

Leaders of Industry is a series of conversations between operational technology, critical infrastructure (CI), and security experts from Armis and other leading companies and institutions. The series explores critical considerations for protecting the OT and CI assets that keep our manufacturing operations, public and private institutions, and cities humming.

This conversation on securing digital cities, campuses, and buildings features the following experts:

• Thomas A. Rodgers, Director of Operational Technology, Penn State University 
• Mirel Sehic, General Manager, Cybersecurity, Honeywell Building Technologies (HBT)
• Keith Walsh, Director, OT Strategy and Operations, Armis

In Part 1, Keith Walsh and Thomas Rodgers discuss how Penn State has addressed security challenges related to connected assets across its facilities that span the Commonwealth of Pennsylvania.

Join us for Part 2 next week as Keith chats with Mirel Sehic From HBT about managing the expanding attack surface in buildings, campuses, and municipalities and how to address them. 

In Part 3, we will outline 5 essential proactive steps for securing our digital buildings, campuses, and cities.

Welcome to the conversation.

Part 1: PSU: An asset Operators View Into the Management of BMS Complexity

Our buildings, campuses, and cities have been growing up fast for the better part of two decades. Today, cities and towns are blanketed with radio frequencies (RF)  to enable our utilities to remotely read gas, water, and electric meters; our highways and bi-ways are covered with smart lighting, cameras, traffic sensors, and speed monitors; and the buildings we work in are networked with devices to ensure safety and comfort to all those who enter.

Just last month, in my not-so-sleepy-anymore-beach-town, I witnessed the construction of 5G poles every 200 meters. With 10x the speed of LTE and significantly lower latencies, we now have a technology that will further accelerate the deployment of industrial IoT devices.

When it comes to finding a dynamic environment illustrating a diverse mix of OT and IoT devices, sensors, and controllers communicating across an ecosystem, one would be hard pressed to find a better example than Penn State University.  With  nearly 2000  buildings hosting 90,000+ in-person and online students Penn State arguably looks more like a smart city than a college campus.

That’s why I was excited to have the opportunity to sit down with Thomas Rodgers, Director of Operational Technology at Penn State to learn more about how he approaches his task of ensuring campuses and buildings maintain their resiliency, safety, and uptime.

Keith: Good morning, Thomas. Can you share with the readers the breadth of the Penn State campus and the challenges you face in the BMS/BAS/smart campus space?

Thomas: Sure. Our BAS systems can be found in many buildings across the Commonwealth of Pennsylvania. Those campuses make up the Facility Automation Systems (FAS). Included in the FAS network are the utility plants for steam/electric generation, wastewater treatment and water treatment facilities, as well as academic buildings, research facilities, museums, dining spaces, and much more. Basically, we are a smart city where most of the building automation systems (BAS) are controlled by the FAS/BAS teams. A big security challenge in the BAS space is visibility into the network and devices. We have many devices on our networks that do not run a traditional operating system where software agents could be installed for endpoint device security posture assessment. In addition, the traditional network scanning vulnerability assessment tools can have a negative effect on OT devices with fragile network stacks.

Another challenge is the scalability and the varying needs of a large and diverse customer base. We are tasked with supporting everything from multimillion dollar research efforts to the simple comfort of heating and cooling. And supporting hundreds of buildings across multiple geographic locations that can be hundreds of miles away provides a unique challenge.  Keeping offsite networks secure and operational at various locations can be difficult. Helping them understand security and the fact that outdated systems need to be updated and replaced at regular intervals is not an easy task and requires us to be constant champions for the cause.

Keith: With such a diverse device ecosystem, how do you wrap your arms around the product life cycle, including patching and updating your various campus support systems?

Thomas: While there are many challenges, applying traditional IT practices to an OT environment isn’t always feasible. The Confidentiality, Integrity, and Availability (CIA) triad in IT gets flipped in OT which presents new priorities in the security space. Performing operating system updates on a server may “break” the building automation software, directly affecting availability. Rigorous testing is required before rolling patches out to your production systems. Also, replacing unsupported operating systems can be challenging because there are multiple parties involved, such as the mechanical systems they are controlling. The building automation software may only be certified to work with a specific operating system and patch level. The endpoint devices may also need to be upgraded or replaced to support the new version of building automation software, which can cost millions of dollars and is not always from the OT budget. Many of these challenges are due to the longevity of our buildings and the extended lifecycle planning of building systems.

Keith: When or how did you realize improvements needed to be made? Or, put another way, what led you to go to market to solve what problems?

Thomas: The most important thing is to understand where you are and create a baseline. In completing our security assessment, in an environment this complex we needed to have a better inventory of authorized/unauthorized devices on the network. The most room for improvement was the visibility of all the software, or devices running on our network. We also lacked a good vulnerability assessment and tracking program, which is key to understanding where you have risk. All of those are key to having a good foundation of cybersecurity in an OT environment. We have a saying on our team, “You don’t know what you don’t know,” and that’s why visibility is key to securing an OT network.

Keith: We couldn’t agree more about needing to know what you don’t know. What might the ideal end state at PSU look like in a perfect world?

Thomas: In the end, an ideal FAS network would have visibility into all aspects of the network and devices. When a new device is attached, a report or alert would go out to the security and network teams letting us know something was connected and needs to be assessed. Routine vulnerability assessments need to be performed by a tool passively scanning the network.

Segmenting the network to separate things like utilities, metering and automation systems is important. Having everything on the same network permits adversaries the ability to pivot from vulnerable systems to other devices.

We all know there is no “perfect world” and there will never be systems without issues or problems, but having a system that is built, secured, and equipped to deal quickly and effectively with the inevitable problems that arise is key to success! We’re well on our way to making that goal a reality.

Keith: That baseline of device discovery you speak of is truly the holy grail because it allows for the successful pursuit of all other use cases, and passive real-time monitoring is certainly the way to go in such a dynamic environment.

Thanks Thomas, we are certainly looking forward to what your journey holds as you progress towards gaining complete visibility of your devices, and your quest to ‘know what you don’t know’.

Thomas: Thanks.

As I look back on my conversation with Thomas, there is certainly good advice to be had. Understanding the environment in which we operate is instrumental. It’s the old saying  – a wise man builds on rock, and a foolish one who builds on sand. As Thomas now has the tools to understand his foundation – the assets, their liabilities, and their interdependencies give PSU a solid foundation to begin layering processes and procedures to reduce the BMS attack surfaces while quickly and effectively dealing with the inevitable problems that will arise in the future.

Join us for Leaders of Industry, Part 2 next week as I chat with Mirel Sehic from Honeywell Building Technologies about managing the expanding attack surface found in buildings, campuses, and municipalities in support of operators such as Thomas and Penn State University.

Additional information on Penn State University can be found at psu.edu.