Always on a Friday

It’s 4:30pm on a Friday. Prime-time for an outage, I say to myself. And right on cue, the phone rings.

I am an engineer for LRS, and I like to keep a close relationship with most of my customers. So I sometimes handle incidents myself and other times hand them off to our support line. In a frantic voice, the customer tells me they have a critical problem. Judging from the urgent tone, simply handing this call off to a colleague in support is not an option; I have to triage the problem myself.

The customer states they have a massive issue with all of their label printers. Somehow the configuration changed on every server and the transformation process LRS uses for labels (which removes the need for a driver) is generating an error. Everyone on the customer’s side is up in arms, all the way up to the CIO.

I tell the customer it’s no problem and that I’ll look at the issue right away. It occurs to me that that during the planning for the implementation, I had requested the company create a vendor account for me. With this account I have the ability to connect via VPN into the customer’s corporate network and have rights to the LRS infrastructure there. I can remote into the servers and find the issue quickly. I soon realize the customer had moved some critical files and directories on the server, and now the path to call the label transform is no longer correct.

So, how do I change over 500 printer configurations all at once? I don't have time to do this manually — not on a Friday afternoon, anyway. As I’m communicating to the customer on their priority 1 bridge line, with a VP and assorted C-level executives listening, it would be embarrassing to announce that I’d have to manually click through 500 print queues.

Luckily, we've been around the block a few times, so LRS has software to handle this exact situation. We can always rely on Murphy rearing his ugly head at the worst possible moment. So we're ready.

I introduce myself on the P1 bridge line as the vendor and tell them I've identified the issue and have a solution. I ask for permission to immediately change each label printer’s configuration so that they point to the new location on the server. Of course, they clear me to make the change right away. Since I still have my remote session open to the server, I'm able to utilize LRS software called PDDT. The real name is Printer Definition and Deployment Tool, but that takes too long to say, much less to explain. So PDDT it is.

With PDDT open, I load the entire printer configuration from the LRS server and identify all of the label printers in their environment. Next, I make the necessary file and directory changes, and apply them to all the label printers. Now I am able to update all of these printer configurations in the LRS server itself. The PDDT software lets me do all of these changes in a spreadsheet style interface. Simple.

Within 5 minutes, I have made a mass change to the customer’s production print environment, correcting all 500 label printer queue configurations. After confirming with the "boots on the ground" that the fix worked, the customer is ecstatic.

I was able to correct an issue that one of their engineers caused and the customer brass thanks me for the quick work. Out of habit, I insist on scheduling a RCA (root cause analysis) meeting for the next week. I hang up the phone, sit back in my chair and breathe a sigh of relief. Let the weekend begin.