As IT professionals, we are constantly updating, modifying, and changing the devices we manage. In this video, you’ll learn how we manage this process with change management.
When you’re making a change to an application or an operating system that you use at home, the scope of that change is usually based on a single computer. But in a corporate environment or a large organization, one single change could affect hundreds or even thousands of different systems. If you’re upgrading software or modifying an application or making a change to a router or a firewall, you’ll need to go through a formal process to make sure that that change is going to work properly.
These types of changes occur constantly. We know that Microsoft has updates for their operating systems every month. And you might have hundreds or even thousands of applications that you use. And those might have constant updates to those as well.
This is something that is incredibly important to stay on top of because a system that is not updated is one that is probably less secure. This is why it’s important to have a formal process for making changes in your environment. If suddenly everyone was able to make any change they’d like whenever they would like, you could run into problems with applications working properly or inconsistencies in the way that applications are able to use the operating system. These processes may dictate how often you’re able to make changes, the type of changes that can be made, and might even cover things like rollback procedures just in case you run into a problem when that change is updated.
Many organizations already have a change control process. And this makes it very easy to implement and control any type of change to your environment. But if your organization doesn’t have a formal change control process, you may find it very difficult to implement or make changes to this corporate culture.
We have these formal change control processes so that we can maintain the uptime and availability of our systems. We know that everyone is informed. So there’s no confusion about the changes being made. And we want to be sure that no mistakes are made when people make changes to these systems.
Here’s a summary of a typical change control process. This might be a little bit different in your environment. But it tends to follow a similar path regardless of where you might be.
The first part of the process is to fill out a formal change control process form. This is something that everyone has to do so that all of the standard information is provided to the central committee that makes these decisions.
You would document in this form the reason that you’re making this change so that everybody understands why this change is occurring. You would then identify the scope of that change. It might be a single system or a number of different systems. And whoever’s making the decision to either allow or disallow this change needs to understand what this scope would really be.
There would be a scheduling process so we would know exactly when the date and time for this change would be. And then we would know what systems would be affected by this change and the impact of that change.
To be able to make a decision on whether this change should occur or not occur, the change control board generally needs to analyze the risk associated with the change. For example, this might be a significant change. And it might be at a time of the year when the company is very busy. So the change control committee would need to balance the risk of not making the change versus implementing the change and having a problem.
At this point, the change control board should have all of the information they need to make a decision on whether the change is allowed or not allowed. And once the change is made, you may want to have users try their systems and confirm that the change was updated without any type of problems.
The change control process usually starts with the owner of the application or the data wanting to make a change to that application or data. The owner of the application or the data don’t generally control the change control process. And they’re not usually the ones making the actual change to that application or to that data.
Instead, the owners are the ones managing the process. The owner is kept informed as the change control process occurs. And once the change is complete, the owner is responsible for testing their systems and verifying that everything is working properly.
For example, your organization may have a department for shipping and receiving. And there may be information that they’ve received that says their address label printers need to be upgraded. In this example, the shipping and receiving department is the owner of this process. And they’re going to hand that off to their IT team to handle the actual change to the printing software.
Another consideration for the change control process are the stakeholders. These are the individuals or departments that will be impacted by the change that you’re proposing. They will probably want to have input on the process and have some type of control over when this particular change occurs.
Identifying the stakeholders may not be as obvious as you might think. There are some changes that can affect a large number of people within the organization, or perhaps the change is only affecting one single individual. The IT team may need to research and identify any of the stakeholders who might be affected by this change.
Let’s take our previous example of upgrading software that’s being used for shipping labels. And you might think that this would only affect the shipping and receiving department. But of course, the shipping labels are also used by accounting to create reports of what has been shipped. There might be product delivery frames affected by this change, especially if you’re shipping directly to your customers. This would, in turn, also affect revenue recognition because it would affect how quickly you’re able to get product into the hands of your customers. And this would certainly have the visibility of the CEO. This is a good example of how something that appears to be a very simple change to some printing software could ultimately have a dramatic effect to the bottom line of the company.
Every change has a different potential impact on the organization. So you need to recognize what risks may be involved when making any particular change. This might be a risk value that you assign a high, medium, or low risk. But these risks could also be very far-reaching.
There may be a case where you install a fix, but it doesn’t actually fix anything. That certainly has a particular risk associated with it, or maybe you install the fix and you end up breaking something else. This is not entirely unheard of when patching systems. There might be a failure of an operating system just because you installed this particular update or there might be corruption of data, which means you would need backups. And all of these would certainly have different levels of risk.
You also have to consider what risks might be involved if you don’t make the change. For example, there might be a security vulnerability and if you don’t patch this application, that vulnerability will be available to any attacker that may be coming across our network. This may cause an application to be unavailable if you don’t patch the application or you might have other services that are no longer operating because you didn’t make a change to a secondary service.
Given these risks, you may decide that you want to do a lot of testing before implementing this change into a production environment. And for that, we may use something like a sandbox testing environment, where you can perform as many tests as you’d like and have no effect on your production systems. This is effectively a technological safe space where you can make mistakes, you can try different techniques, and then you can perform extensive testing after the fact to confirm that the update really did work properly.
Inside of the sandbox, we can load a duplicate environment to our production systems. And then we can try the upgrade, apply a patch, make the change, and see what effect that has to a system that is identical to what your users are using in production.
This is also a good time to test your contingency plan. You can implement the change. And even if the change was operational in your test environment, you may want to test your backout procedures just to make sure that if something does go wrong in production, you have a documented series of steps that brings everything back to the way it was before you made the change.
There are many documented cases through the years where someone thought that a change was very minor, they make that change into an environment without even thinking that they would need to be able to revert back to a previous config, and that change ends up bringing down their entire network. This is why you need a backout plan. You should have a way to revert back to the original configuration or something that would be very similar to what you started with before the change took place.
In some cases, this is a very simple process. You simply uninstall the patch that you installed and confirm that the original files are back in place. But some changes are very difficult to revert. So you may need to have different techniques and ideas about how you can bring those systems back to their original form if something does go wrong.
This is why we often say that before you make any change to a system– that you have a full and complete backup of that system. That way, if you do make the change and run into a problem and then you try your backout plan and have a problem, you can always go back to your backup.
You may find that the process for approval of the change is the easy part. The difficult part is finding time to implement the change. This is something that has to be planned and considered before making any significant change to production.
For example, you may not want to make change during the workday, when everybody is on their systems and performing their work. Instead, you may want to have off hours or maintenance hours where you can make changes without having significant impact to users. This is why you often see the IT folks coming in on weekends and holidays and very early in the morning just so they can make change without any type of disruption to the network. This can be especially challenging if you work in an environment that is 24 hours a day and seven days a week, where you have very little time available to make any changes in those systems.
You might also need to consider what time of the year you’re making these changes. Take, for example, a company that is very retail-based and their busiest times of the year are between Thanksgiving and New Year’s. In those environments, it’s not uncommon for all of their systems to be completely frozen during that time frame. And no changes would be allowed whatsoever. Only after the new year are you now able to reintroduce some type of change control and begin the process of updating your systems.
Hopefully, you can see now that change management is a critical part of your policies and procedures for security. And it affects every single person in your organization. In almost every environment, this change control process is well documented, and anyone in the company can read through the documentation on their intranet. This should be part of your standard operating procedures. And no one should be able to make changes on your network without receiving this approval. And of course, your change control process is going to be updated over time to make it more efficient and fit the best with the requirements of the company.