Boutique vulnerabilities (BioPhorum cyber security education series: one)

The BioPhorum IT Cyber Security Education Series: 1

This is the first in an ongoing series of publications to be delivered by the BioPhorum IT Cyber Workstream as part of the new BioPhorum IT Cyber Security Education Series– a portfolio of articles and short papers addressing emerging issues in cyber security in the biopharmaceutical manufacturing environment. Focusing on key findings from extensive surveys, case studies and peer reviews, the series will provide valuable information about cyber security issues and challenges experienced by BioPhorum members and comparing and sharing how members are responding to these in today’s fast-moving environment. The intention is that the series will form the basis for a new set of BioPhorum guiding standards and best practice principles for cyber security in biopharmaceutical manufacturing.

Improving performance against boutique vulnerabilities

Introduction

Manufacturing organizations today face an increasing number of new cyber security threats, and many require urgent action to shore up defenses, typically but not exclusively through patching. Several of these so-called ‘boutique vulnerabilities’ have appeared in recent years – WannaCry, BlueKeep, DejaBlue to name a few – and each has required unplanned, near-term action to address.

These efforts are disruptive, diverting IT (enterprise information technology) and OT (operational technology/process control) resources away from other work and resulting in ad hocimpacts to manufacturing, as equipment outages are required to complete patching and related work.

With each new challenge, organizations are working to improve their response and reduce the time and effort required. Companies are all investing in routine patching where possible, reducing this activity in urgent situations, as well as investing in other mitigation options such as isolation where appropriate – more options means reduced impact. Across organizations everyone, including quality, manufacturing, external application vendors, and senior leadership is working to improve understanding and ensure support is in place in advance.

New vulnerabilities demand quick mobilization of resources

While new vulnerabilities are identified all the time, some rise to attention because of the potential severity of the consequences. These are sometimes called ‘boutique vulnerabilities. Here are some well-known examples which have arisen since 2018[1]:

  • WannaCry appeared in May 2017 and targeted an early version of the Server Message Block (SMB) protocol used by Windows for mapping drives between machines. Microsoft took the unusual, and therefore noteworthy, approach of issuing a patch for no-longer-supported versions of Windows, including Windows 2003 and Windows XP. In addition to patching, this vulnerability could be mitigated by disabling the SMB v1 protocol using registry entries.
  • BlueKeep appeared in May 2018 and targeted the Remote Desktop Service (RDS) on versions of Windows Server including Windows 2008 and the no-longer-supported Windows 2003. Microsoft again provided patches for Windows 2003 servers. The vulnerability could be mitigated by patching or by disabling RDS. This vulnerability raised concerns because RDS is commonly used for support and no credentials were required to exploit it and potentially run attack code freely.
  • DejaBlue appeared later in 2018, like BlueKeep but affecting Windows 2008 and newer versions of Windows Server. Again, mitigation options included both patching and disabling of RDS.

In each case, there was a concern that an attacker would have little trouble leveraging the vulnerability once they could physically reach the machine. The fact that drive mapping and RDS were involved, both using protocols that are often part of normal communications through firewalls, meant that doors were open that had to be closed quickly. In short, if network defenses failed or were overcome, the vulnerabilities were easy to exploit, and the consequences could be significant.

Member companies all felt compelled to mount an immediate effort to patch or otherwise protect as many systems as possible in their manufacturing operations.

Efforts are time consuming and disruptive

Mobilizing IT/OT staff at manufacturing sites and in support roles across the company is time consuming. Manufacturing systems are also affected, so local quality groups, supporting software and automation vendors, as well as manufacturing managers are also involved. Although time consuming, it is essential for all involved to:

  • Identify vulnerable assets

Sometimes this is just a question of the operating system version, but there could be other criteria like using a specific service, or you may exclude assets that are isolated or otherwise protected. 

  • Work with system vendors who are ideally vetting patches on your behalf

Many systems are supported by application or system vendors, and often these vendors provide testing of new patches with their software andalsocondition their ongoing support on the basis you will not modify the system without coordinating with them. 

  • Work with internal quality teams to manage and document changes required to patch or otherwise update systems

Updates are considered a change, so they require risk assessments, tracking by quality teams, and verification of proper operation after they have been applied. 

  • Work with manufacturing teams to secure outages for applying patches

Production equipment cannot be rebooted without close coordination with manufacturing, typically requiring an outage between batches.

These efforts are disruptive because they are urgent, frequently pushing other efforts aside, for example by:

  • Pulling IT/OT staff off other work

Work that is potentially of higher strategic value is put on hold while mitigation efforts run their course.

  • Disrupting manufacturing by requiring outage windows between batches

Even waiting until batches are complete before fitting a patch disrupts the planned rhythm of the operation.

  • Scheduling of outages further in advance is difficult

Vendors provide valuable testing of patches, but delays and uncertainties make proactive scheduling of outages impractical and unlikely.

BioPhorum member companies have demonstrated numerous optimizations

A member company provided a short overview of metrics from their WannaCry, BlueKeep, and DejaBlue efforts. Here are some of the key points:

  • This isn’t easy.

For WannaCry, the first effort, the member company managed to fully protect over 80% of vulnerable assets within roughly 40 days, and all vulnerable assets only after 13 months.

  • Having options beyond just patching makes a measurable difference.

The BlueKeep effort benefited from lessons learned, and all vulnerable assets were protected within only 22 days. A key difference for BlueKeep was that assets which could not be patched, typically because of manufacturing scheduling, could still be protectedbecause there was an option to just disable RDS until those systems could be patched later. Some assets that had been identified for upgrade or retirement were retired immediately, and others were immediately isolated.

  • Constant improvement is sometimes limited by factors outside your control.

The later DejaBlue effort seemed easier because the assets ran newer versions of Windows. However, in reality it was harder because so many had not been patched and required lengthy periods to complete numerous accumulated patches to reach ‘current’. Vendors also identified some patch conflicts and dependencies[2]. 80% of the assets were protected by day 40, although some remained unprotected beyond day 90.

Discussion among member companies revealed similar findings and experiences were identified across different companies. These findings are summarized below:

  • Track and manage efforts using Excel, databases, CMDB, or a combination.

However, everyone wants to move away from spreadsheets and are investigating technology options for support.

  • Working directly with manufacturing is key.

This is the ‘new normal’. It is important for all stakeholders to understand this and to have a sense of the criticality of getting this work done.

  • Top-down leadership helps, a lot.

Urgent, disruptive, labor-intensive efforts require broad cooperation among various groups and quick, temporary realignment of priorities, all of which require executive support during as well as after an effort.

  • In addition to frequent patching, supporting software must be current too.

During the DejaBlue effort, older versions of McAfee Antivirus and Microsoft SCCM created challenges for everyone.

  • Use a ‘patch-to-endpoint’ strategy during an urgent effort.

Some companies go ahead and install patches aggressively, and thenwork with manufacturing to plan the reboots that will actually apply the changes. Some found this does not work well for routine patching because the reboots are a long time coming and if multiple patches are pending, things are less predictable from a time-required perspective. However, there was general agreement this is a good strategy during an urgent effort.

What are your peers doing?

The responses to these boutique vulnerabilities are still disruptive, ad hocefforts, but everyone is learning and applying those lessons. Everyone has included the following activities in their toolkit, in some form:

  • Improving communications to stakeholders.

We need to tell our stories very clearly. For example, it is reasonable for people outside the actual work to assume that each new effort will set a new ‘worst case’ in terms of how long it takes to address these things. In practice, however, this is not necessarily the case and there are always new unknowns and things outside of our control. Every set of new vulnerabilities is unique.

  • Leveraging all options, not just patching, to protect individual assets.

There are often temporary mitigations like disabling a specific OS feature that can protect assets more quickly and with less impact than patching. This makes it possible to schedule subsequent patching to permanently close vulnerabilities when the business impact is smaller.

  • Learning that everything is not Windows.

All of the examples here were Windows vulnerabilities, but there are other assets out there. Recently, the URGENT/11 vulnerability was identified, impacting the VxWorx operating system at the heart of many industrial control devices. We can’t limit our planning to Windows vulnerabilities.

  • Adopting an approach of constant, incremental improvements.

Your peers are thinking about process improvements for the next effort while they are closing out the current effort. Post-effort reviews highlight potential improvements in tracking tools, asset inventory, and procedures. While these vulnerabilities appear without warning, the period immediately following an effort is a good time to make improvements.

  • Pursuing increased routine patching as a means to reduce the impact of these ad hoc efforts.

If, for example, 40% of your systems will get patching within the next 60 days through routine patching, you can decide whether to move faster when a new critical vulnerability appears or just let the routine patching play out, focusing your ad hoceffort on the gaps.

[1]Why isn’t NotPetya, the 2017 attack that so seriously impacted Merck, Maersk, and other organizations around the world, included in this list? Companies that were vulnerable had either patched the specific targets or not, and either used the infected software or not. Though a patch was issued several months earlier, it was not a situation where a new and frightening vulnerability was announced to the world leading to a race between attackers and defenders.

[2]These problems affected all members equally, based on the discussion. Vendors like Microsoft and Emerson were aggressive in identifying and addressing these kinds of problems, and almost everyone had to closely follow those developments as they occurred.

Follow BioPhorum

Related tags

Recent releases

0 Comments

Submit a Comment

Your email address will not be published. Required fields are marked *

Cell & Gene Therapy

Maximizing the impact of the many advanced therapies under development requires that the industry comes together to improve understanding of how to develop, control and manufacture these life changing therapies.

We connect strategic thinkers in drug development and contract manufacturing organizations with the aim of ensuring harmonization and alignment around issues such as potency assays, phase-appropriate guidance for critical quality attributes, operator safety, regulatory guidelines, and C&GT-specific validation issues, as well as working toward resolving the current challenges to commercializing C&GT products. Discover more and learn about the outputs here.

Development Group

The ever-increasing need to reduce product development lead times and the opportunities to exploit innovative technologies in development are two key industry themes that remain at the forefront of our work. The Phorum successfully connects process development organizations and provides a space where subject matter experts can work on issues such as quality by design and process characterization, cell line technologies, and molecule selection. Read some of the outputs and discover how you could benefit by getting involved.

Drug Substance

Accelerating improvement, exploiting opportunities offered by innovation, and identifying best practice approaches across all aspects of the manufacturing process – these are the essential objectives

Data and digital capabilities are at the core of many of these industry challenges. Work in BioPhorum Drug Substance is informed by BioPhorum Technology Strategy, which has summarized our industry’s market drivers and trends. Capacity, flexibility, sustainability and pace all feature as requirements to succeed in a rapidly changing environment.

Find out more about what our members are currently discussing and learn about some of the successful implementations of our output.

Fill Finish

Quality and demonstrating compliance are two priorities of BioPhorum Fill Finish. Representatives of the world’s top sterile filling operations come together to develop solutions to some of the most intractable technical and regulatory challenges.To address these challenges in drug product aseptic filling (‘fill’) and secondary packaging (‘finish’) operations, our community of industry experts is working relentlessly to continually improve the pharmaceutical quality system (PQS). Together they continue to strive towards developing and implementing safe, predictable, lean, and agile processes in drug product operations. Find out more about the Phorum output here. 

Information Technology

IT and digital advances are crucial in responding to the pharmaceutical industry’s challenges around speed, cost, quality, flexibility, and sustainability. Unlocking and realizing the value in the industry’s IT opportunities is the core mission of the IT Phorum. 

The digital manufacturing facility of the future needs to consider technology advances alongside organizational perspectives while maintaining foundational security requirements. All elements must be considered in the long term to deliver systemic change toward the digital factory vision. Our members collaborate in a format that stimulates and supports industry innovation and implementation. 

Regulatory

BioPhorum Regulatory CMC is affording professionals the opportunity to shape their own collaboration program focused on challenges and opportunities across strategy, science, operations, submissions and compliance. Following a discovery program of work with Heads of Regulatory CMC, in which we identified some key topics of collaboration, we are giving you the opportunity to participate and help us to shape the final collaboration program. This will focus on, Filing and Review, Interactions with Health Authorities, and Harmonization. Our mission is not just to discuss, but to deliver applicable outputs and publications that will have an impact – learn more about how you can get involved.

Supply Chain to Patient

The pharmaceutical outbound supply chain was built on a blockbuster model and doesn’t fit the needs of the changing pipeline and new modalities currently being developed. The geopolitical challenges of recent years have shown that the outbound supply chain needs to evolve to become more agile, and resilient, and address a more expectant end-user bying on patient-centricity. Industry collaboration can accelerate improvements and transform performance in a way not possible when working independently. This Phorum is founded on the belief that learning from each other and challenging current thinking to re-imagine the supply chain can reduce costs and increase efficiency.

Supply Partner

BioPhorum SP members want the inbound supply chain for the biomanufacturing industry to be recognized as one of the best-understood and managed global supply chains. They intend to be not only a trusted advisory group but also the engine that drives change for the industry. Working together, we aim to achieve supply chain transformation that provides:

  • an enhanced robustness and resilience
  • accelerated capability and capacity to effect technology adoption
  • reduced cost of quality, and
  • industry-wide alignment on operational implementation

There’s an extensive list of Phorum successes and tangible benefits that have been delivered to define industry best practise.

Sustainability

The BioPhorum Sustainability program is jointly designed, governed, and delivered by a membership drawn from licence holders, contract manufacturers, major suppliers and niche suppliers. This dynamic partnership means that we can quickly focus on the areas that have the greatest industry impact, and we can leverage learning from each other and other sectors – the chemical industry and the built environment for example.

BioPhorum’s unique ability to build a strong coalition for change for an entire industry is now being applied to the greatest challenge of our generation. Driven by evident industry needs, Phorum members are working to develop agreed industry benchmarks, establish the ROI of sustainability investments and to increase the efficacy of implementation of members’ sustainability strategies.

Technology Strategy

With a mission to define strategies to accelerate technology transformation and support the adoption of innovative technologies, BioPhorum Technology Strategy evolved from BioPhorum Technology Roadmapping, strives to realize the technology roadmap vision 2.0.

This is being achieved through a multi-stakeholder team of member companies with collective capability and influence who are committed to driving forth industry goals associated with: Speed to market, Quality, Value, Agility and On-Demand Supply and Sustainability.

Building on industry feedback to standardize plasmid release specifications

Plasmids release specifications are critical to the manufacture of many cell and gene therapy (CGT) products, but current regulatory... read more

How to predict, optimize and analyze high-concentration biologic therapeutic formulations

Developing high-dose biologic drugs for subcutaneous injection often requires high-concentration formulations. It also needs to optimize... read more

A media fingerprinting toolbox that comes with a host of benefits

Cell culture media raw materials can contain many chemical substances, each with its own properties. While simple substances can be... read more

Why 100% visual inspection does not mean 100% defect detection

There is a common misunderstanding that 100% visual inspection (VI) of biopharmaceutical products should mean 100% detection of defects.... read more

How to bridge the data integration gap between sponsors and contract organizations

Bringing BioPhorum’s unique perspective to continuous manufacturing in biologics

Continuous manufacturing (CM) has been a goal for the pharmaceutical industry for several years. While small molecule drug substances and... read more

Supply Chain to Patient has launched!

Have your outbound supply chain costs increased? Do you want to leverage data and digital technologies more effectively to reduce... read more

Electronic data exchange – the foundation for an efficient, resilient inbound supply chain

The COVID-19 pandemic, geopolitical unrest, and military action across the world have fundamentally disrupted the assumption and practice... read more

How are you improving your environmental sustainability performance?

Climate change and global warming resulting from greenhouse gas emissions are widely recognized as the biggest threats to global health.... read more

Adding value for analytical instrument partners

Analytical instrument vendors have been integral to developing and publishing our technology roadmaps and developing product... read more

Introducing MediPhorum

The medical technology (MedTech) industry is a critical and complex sector. As highlighted during the Covid-19 pandemic, the MedTech industry provides a wide array of products and technologies that are vital to the safe and effective provision of care to many. Also, as seen over recent years, there is often the requirement for flexibility in supply, rapid development, and the introduction of new products to keep pace with global public health demands.

Advanced Wound Care and Wound Management

Advanced wound care (AWC) is the medical treatment for wounds that have not healed properly using conventional wound care products such as gauze, bandages, plasters, and wadding.

AWC involves treating chronic or complex wounds using products that are not as easily obtained over the counter. AWC can be divided into three main categories: wound healing, infection prevention, and restoration.

Drug Delivery

Drug delivery products are therapeutic and diagnostic products combining drugs, devices, and/or biological products, which enable safer and more efficient treatment due to careful and precise drug targeting, local administration, and tailored treatment.

Introducing the new MediPhorum

MediPhorum by BioPhorum has been launched to bring together senior business leaders and subject matter experts across multiple MedTech... read more

Introducing the new MediPhorum

MediPhorum by BioPhorum has been launched to bring together senior business leaders and subject matter experts across multiple MedTech... read more

Introducing the new MediPhorum

MediPhorum by BioPhorum has been launched to bring together senior business leaders and subject matter experts across multiple MedTech... read more
BioPhorum Deliverables Report
BioPhorum brings you the latest Deliverables Report, an informative and inspiring consolidation of the benchmarks, position papers, guidance documents, assessment tools, industry user requirements, study results, and roadmaps produced in the last twelve months.
BioPhorum Downloads
Access a variety of free tools and papers and other resources designed to provide you with up-to-date information and insights to help you make informed decisions and maximize your success. Access our resources today and start making improving and streamlining your business processes.
Case Studies
Welcome to the BioPhorum case studies page, where you can learn how our clients have used our service to discover innovative solutions and save money. Here you can read real-life stories of how various organizations have used our unique data-driven approach to streamline their processes and make the most of their resources. Whether you're looking for inspiration or want to learn more about the potential benefits of our service, you'll find it here.
Webinars and podcasts
Welcome to BioPhorum Connect, the podcast series that keeps you up to date with the latest news and trends in the biopharmaceutical industry. From experienced professionals to thought leaders, this podcast brings you the insights and perspectives of experts from around the world. We'll discuss all aspects of the biopharmaceutical industry, from digitization and knowledge management to raw materials, supply chain, sustainability and more. So, join us on our journey as we explore the world of biopharmaceuticals and learn from the perspectives of those at the forefront of the industry.
Share This