In Stay Sharp Episode 153, Zero Downtime, No Disruptions: Honeywell’s MES Upgrade Story, co-hosts Juliann Grant and Jonathan Scott dive into the success story of Honeywell’s MES upgrade with two key players: Ryan Sims, Senior IT Engineer for Honeywell, and Brian Neff, Technical Team Lead for Razorleaf. Ryan and Brian’s insight into what worked well and what didn’t leads to simple, clear advice for everyone facing their own upgrades.
What the Upgrade Needed to Accomplish
Ryan explains that the Honeywell manufacturing site in Clearwater, Florida, originally moved to iBase-t’s Solumina MES to replace two legacy tech-debt MES solutions. This version upgrade was meant to stay ahead of the tech-debt trap, while minimizing disruption and risk. Ryan says, “Today’s deferred maintenance is tomorrow’s tech debt, so we wanted to prevent Solumina getting into that tech debt situation that our previous MESs had gotten into.” But the fundamental goal was making sure there was absolutely no disruption to the shop floor.
To do so, they upgraded to a newer version of Solumina (i110), upgraded their Linux operating system, and provisioned new servers to host the Kubernetes cluster running the application. Because they had new application servers, they were able to stand up the entire new infrastructure in parallel to the old version that was still live—giving them a blue-green environment that was key to the upgrade strategy. After setting up the new system, the team took a database snapshot as a backup, ran the database update scripts, and changed the reverse proxy server configuration to point to the new primary node of the new Kubernetes cluster.
Ryan Sims, Senior IT Engineer, Honeywell
Brian Neff, Technical Team Lead, Razorleaf
Because of the reverse proxy server, the URL people were using to access the application didn’t change. Ryan explains, “This is instantaneous cutting over. You change the URL rewrite, and you restart Apache.” Jonathan praises the blue-green approach, saying, “The new system and the old system running in parallel and being able to switch over during cutover instead of let’s build it up and cross our fingers every step along the way.” The relative simplicity of the upgrade strategy meant that they were well-prepared in case they needed to roll back.
Learn More About Honeywell’s Successful Upgrade Process
Another piece of their successful upgrade strategy was leveraging Honeywell’s numerous non-production IT environments of Solumina throughout the upgrade process. They specifically didn’t upgrade the QA environment so they had a previous version intact to test any issues that might come up after the cutover. But in their dev, test, and sandbox environments, they iteratively tested and updated the upgrade scripts. They also installed Solumina on their new servers weeks ahead of go-live, connecting them to Honeywell’s non-production communication servers to test the configurations. The ability to gauge how long the upgrade would take was critical. Brian says, “Utilizing those environments allowed us to time this thing out so we can know where we’re going to be at in the overall progress of the go-live day.”
Lessons Learned
Ryan describes two issues that came up during the upgrade that could have been disastrous if their planning hadn’t been so thorough. First, an upstream internet connection in Florida that should have failed over to a backup didn’t, so he and others running the database upgrade remotely lost the connection in the middle of the upgrade. When the connection was restored, they discovered the scripts were still running locally, though starting to fail due to directory overload. Because of their thorough preparation, they quickly did some cleanup and database reconfiguration, restored the database backup, and re-initiated the upgrade scripts. He concludes, “It really drives home the importance of having that robust rollback plan because if we had not taken that full backup of the database, we would’ve been in a world of hurt.”
Testing was paramount, along with having a robust rollback plan. Ryan and team were able to perform admin-level testing of the upgrade process in non-production environments, which also hosted user acceptance training, before the actual upgrade happened. “We’re continually upgrading our UAT test plans,” Brian says. “I think it’s one of the most important things that allowed us to help define a go, no-go criteria as well.” Ryan agrees, “Make sure that your test plans are kept up to date and constantly be thinking about what more can we be adding to our test scripts. If you think you’ve tested it enough, you aren’t thinking hard enough.”
The full podcast contains more discussion and details on a variety of topics, including why Ryan referred to “Schrödinger’s migration scripts,” how end users were able to test their SAP integrations to Solumina prior to the actual upgrade, how smoke testing immediately after the cutover exposed a problem that needed a quick fix, and more.
Be sure to check out Stay Sharp Episode 153: Zero Downtime, No Disruptions: Honeywell’s MES Upgrade Story, and join us each week for a new podcast.
Follow the Razorleaf Podcast, Stay Sharp in Digital Engineering on:
