Guest: Matt Cramer, DevOps Engineer, ASRC Federal
Episode Description
As an IT professional in the public sector, ASRC Federal DevOps Engineer Matt Cramer dives into legacy IT systems in the modern workplace. From upgrading them to fit within changing environments to spearheading rip-and-replace overhauls, Matt and podcast host Ryan Bacon discuss their experience with outdated tech and what it takes to modernize an environment using legacy equipment. They cover the challenges of working with legacy technology, trends they鈥檙e seeing as the modern workplace moves away from premised systems, and how to effect change collaboratively with users, stakeholders, and leaders.
This episode of Where鈥檚 the Any Key? offers IT professionals insights into approaching problematic legacy systems and what鈥檚 involved in driving, fostering, mitigating, and shaping IT change.
Introducing Matt Cramer
Ryan Bacon: Welcome to Where’s The Any Key?, the podcast where we talk shop about topics, tips, and trends for the modern IT admin. I’m your host, Ryan Bacon, the IT support manager at 黑料海角91入口. Joining me today is Matt Cramer. He’s a DevOps engineer at ASRC Federal. Thanks for coming on, Matt.
Matt Cramer: Thanks for having me. I really appreciate it.
Ryan: Oh, sure thing. Why don’t you tell us a little bit about yourself?
Matt: So I have a long and varied past that has touched on technology for quite a while. I was actually just thinking about this last week. I think my first encounters with technology were back in the 1980s in elementary school. We all had the brand new shiny Apple IIe computer labs, and we played our five-and-a-half-inch floppy disc, Oregon Trail. And then when I was in middle school, I got introduced to some early DOS. I had a friend give me a little IBM 286 DOS machine and no instruction manual. So I got to just go hacking around in it to try to figure out how CLI systems worked. And then over the years, I sort of just tinkered with technology along with a whole bunch of other manual trades. I worked in the nursery business, moving plants around. I worked in construction. I’ve done a whole lot of things and I share that because it all becomes relevant to where I’m at today, later on.
But I really got involved heavily in technology, probably in 2010, 2011, and then really ramped up the following year in 2012 when I started doing some tech purchasing for a brand new charter school that my wife was actually starting. So I had to meet with tech reps and figure out how to be knowledgeable about what equipment to purchase for this new school. And then in the years that followed, I ended up working as a systems administrator in another school, out here in Greeley, Colorado. Spent seven years there doing systems administration, network engineering, deskside support, you name it. I was one of a very small team. There were about 1,000 students and we had four people on the IT staff, two full-time, two part-time, that covered all of that operation. So I had to do it all and learn to do it all. And then a little over a year ago, I made a move into the DevOps space where I’ve been working for ASRC Federal, mostly doing CI/CD pipeline work, automation, integrating automated testing strategies, and reporting schemes for those. So there it is in a nutshell.
Ryan: Yeah, that’s a good varied background. You touched on a lot of different aspects of IT there. That’s really neat.
Matt: Yeah. And this is a theme that I’ve heard come up in the podcast before about IT generalists and I like that term in that a generalist is able to do a lot of things, but may not be necessarily an expert in any one particular niche, but that’s not a bad thing. And I think for a long time, I thought that that was a bad thing. And like, if I’m not an expert in something, if I can’t put that one thing on my resume that’s the real powerhouse, then I’ve got a deficient resume or deficient work experience. And that’s definitely not proved to be the case in my case. And I’ve seen it a lot. A lot of people just coming from wide backgrounds bring a really varied perspective to solving problems.
Ryan: Yeah. And especially when you’re working as the only person on a team or on a small team, you need to be able to do a lot. I would hate to be someone who is a dedicated network engineer try to fill the role of a one-person IT shop.
Matt: Sure.
Handling Legacy Policies and Practices
Ryan: You’ll learn. You’ll have to learn. But I feel it’s starting off as a disadvantage. But I do want to get into our topic because I think it’s a really good one that a lot of people, not only the IT field, but in a lot of industries, people filling a lot of roles can commiserate with us. And that is addressing legacy policies. This could be hardware. This could be, like I said, policy or software procedures, that sort of thing. I’m going to hand it off to you to kick off your thoughts on maybe some experiences with handling legacy policies and practices.
Matt: Sure. So I’ll give you a real relevant one that I’m dealing with right now. And in fact, I just came out of a meeting where we were discussing moving some very, very legacy technology into a more modern place. And this is where you have to address both some creative thinking, some problem solving, some strategy, and then some real hard work because it involves touching people. It involves touching personalities and other people’s pet peeves and other people’s priorities. So moving legacy systems, whether that be policies and procedures or whether that be software, whether that be operating systems, whatever it is, you have to first address the people in the mindset of the team that is being touched by this. I think until you can get team buy-in, it’s very, very hard to move the needle anywhere. And getting team buy-in is probably the most delicate thing that we have to navigate, right?
It’s very easy to, say, come in as a big boss and say, “all right, we’re going to move this system from point A to point B, and here’s how you’re going to do it, here’s when you’re going to do it and here’s what it’s going to look like when it’s done.” And people may or may not be happy with that decision, but since they’re the ones who have to do the work, they’re the boots on the ground, it’s really, really important to get them to buy into the vision of why we’re doing this. There’s a great principle out there. I think it was Simon Sinek who has a book I think called 鈥淪tart With Why.鈥� And if you can’t get people to buy into the why, it’s very difficult to get them to accomplish the how.
I really believe collaborative mission and vision is really, really critical to team buy-in and team success. Get everybody together and really just get everybody’s input. You’ll get a baseline temperature for the room and see what everybody feels about this. And then that really starts to reveal some things that, maybe as a team leader you didn’t perhaps think about initially in the process, that you missed, and having other people’s eyes on it and sort of picking it up and looking at the Rubik’s cube from different angles really reveals more than what one person could bring to the table.
Ryan: Yeah, definitely. You bring up a good point with that. So one thing that I experienced here at 黑料海角91入口, I was the first IT hire, and for a long while I was working as a one-man shop. When the team started growing, when we brought the second person on, then the third person on. Any time you do that, you have your processes scrutinized and reassessed with a fresh pair of eyes. And when it’s just you, you don’t necessarily have to explain it to a lot of other people. And also, situations change. What made sense to me two years ago may not only not make sense to the other person on the team, but it may not make sense to the situation that the organization’s currently in.
And now that I’m in a leadership position on the team, it’s, first off, being able to accept that and be willing to look at stuff that I did in the past and be like, either, one, I was wrong – which is very possible because I’m human – or this is the wrong answer for now. A good example that I can think of actually happened this morning with a conversation with me and another member of the team where he’s going through and updating our documentation and stuff like that. I mean, when you do that, that’s a great time to catch these legacy artifacts from the past. And he was going over how we reimage systems and stuff like that.
And the instructions that we had in our playbook was, you know, “go in there, do a full wipe along with an overwrite” which is a practice that’s leftover from mechanical hard drives and even early SSDs, but with modern SSDs, it’s a waste of time because it’s not necessary. It was a quick conversation. Easy decision, just remove that set of instructions from the playbook, but I think that it’s a good example. You should be going back and regularly reassessing your processes because things change, best practices change and you don’t want to get stuck in a way of doing things just because that’s just how you’ve done it.
Matt: Right. Yeah. I just had one of the Dev leads on one of our teams send me sort of a chuckle this week with a screenshot of the same kind of documentation that documented the process of installing one of our main APIs. And it was hold-over documentation from goodness knows how long ago that involved remote desktop onto a machine, add, remove programs, uninstall, copy the new package over, install, manual verify that configs are correct, things like that. And we just laughed because the tool that is replacing it, that this documentation was being rewritten for, is a brand new API that builds its own hosting environment as it installs. It’s just part of the process as we stand it up. And so it was really, really funny to see that.
And I think it was one of those things where the transformation of the old legacy process to the potential for new processes was in stark relief because we saw how things had been done. And we know because we’ve all done those things for years and years and years that when we see the new one, like people’s minds are going to be blown on the production deployment teams when they see this. And in fact, they already are in a lot of ways. I’ve been in a shop, mostly, .NET Framework applications, and they’re slowly converting to .NET Core so that we have the potential to run them in containers on Linux hosts. So trying to move that needle a little bit, but it’s hard to meet the demands of the customer, new feature development, feature improvement, efficiency improvements, things like that, along with transforming an application for modernization. But this particular one was actually a ground-up rebuild. It’s going to start from scratch. We’re going to stand it up side by side and then phase the old one out eventually and cut it off rather than try to refactor it because it’s a behemoth.
And so you can’t always do that, but in this case, that was the best answer. And so, moving the production team from the old style to the new style has been both challenging and entertaining at the same time because they’re making requests now in the legacy way for deployment. Oh, we need an MSI for this particular environment deployed to our Artifactory so that we can pull it down and install it. And five or six times a release cycle, we have to remind them, oh, no, those are environment-agnostic now. There’s just one package. All you do is you go push the deploy button into your target host and it stands itself up. And they don’t quite know what to do with that because it’s too easy and it takes seconds to install these new applications, whereas before it could be several minutes.
The Evolving User Experience
Ryan: Yeah. And the too easy part is a really interesting thing that I’ve noticed when moving to new systems. So we moved to a new help desk system here about a year and a half ago, and we did it to make things easier for our coworkers at 黑料海角91入口 to reach out to us, to communicate with us. As you alluded to, buy-in and adoption of a new process is critical in order for you to fully leverage the benefits of it. And one of the things with help desks is when you get these requests, people will go through whatever channel’s easiest for them to get these requests in even if you’re like, 鈥淥kay, you need to log into the help desk and fill out this form,鈥� or whatever.
Most of our communication happens within Slack. So our new help desk, we can take requests, whether it’s a DM, whether it’s in our IT support channel, anything like that, and convert that message, with a single or a couple of clicks of the button, over into a ticket and the person, their interface with the whole ticketing system is just through Slack. And an interesting thing that I noticed was when we rolled this out, we said, okay, at this point you don’t have to worry about if something needs to be a ticket or not. You don’t have to worry about that. That’s all on us. Just tell us how we can help you.
Tell us what you’re looking for and then we can make that decision and convert it into a ticket if needed. We’ll do all the work for you. You just communicate with us normally and all this stuff. And we still get a lot of people who are like, 鈥淗ow do I send you a ticket?鈥� Like, 鈥淒oes this need to be a ticket?鈥� I feel that in this case, in particular, experience with traditional ticketing systems or more accurately, traditional IT practices have ingrained that dealing with IT is going to be jumping through hoops, is going to be, you have to follow this set procedure to get the assistance that you need. And changing that and making it super accessible, super easy, people weren’t quite ready for that. It’s taken some time for some people to, I guess, come to terms with the new normal being easier than it has been in the past.
Matt: Yeah. That’s a really good point. I like that example actually. In our ecosystem – I work for ASRC Federal, which is a company that serves government contracts for software development, so my particular division is all doing agricultural software – and on our greater business unit, there are a number of different vendors who are all doing development on different pieces of the giant puzzle. And my team is positioned such that we are providing, basically, pipeline as a service to all of these vendors. And the biggest mind-blower for people and probably the biggest obstacle for the development side is to understand that they no longer have to go through all of those little custom steps to build and get the software ready for a certain state before it can be put on a server.
Or they’ve got to manually log into a server in order to fiddle with a file that got installed there, that they’ve got to update a connection string for a database or something like that. They’ve got to do it afterwards. Whereas now with this new mindset, we call it pipeline as a service, all they have to do is commit code to the correct branch. You write code, you commit it to the Git repository and that triggers a series of events and the pipeline picks it up and just puts it where it needs to go. And they don’t sometimes quite know what to do with that because it’s too easy, to that point you just brought up.
Establishing Buy-In
Ryan Bacon: Yeah. I’m curious to get your experience and get your view with dealing with all these different vendors collaborating on a product or on a process. So you’re not only dealing with your own organization’s legacy processes, you’re dealing with the same for all these vendors. What’s it like trying to get everybody onto the same page with new processes and that sort of thing?
Matt: Well, it’s a two-step approach, and one of them we briefly touched on earlier about mission and vision. Talking with the people, getting buy-in on why. Why are we doing this? What’s the objective? Part of it is also driven by the fact that we are in a government contract relationship. So the needs of our customer, and this is where it’s weird, we only have one customer, but their needs are paramount. And so we have to then come back and collaborate with them to prioritize which things get done when, but as far as getting the different contractors to buy in, there’s that layer of mission, vision, buy-in and then there’s a layer of, we call them solution-level objectives. Big picture goals that we’re all shooting at.
As large as this organization – organizational structure, I’ll say, because it’s multiple organizations – as large as the structure is, we’ve adopted a pretty strict scaled agile framework at the enterprise or solution level scale. So multiple release trains, multiple product owners, and they have to all collaborate in their own way. And everything sort of trickles across. Those two things may be somewhat counterintuitive because agile in and of itself is designed to sort of be flexible and fluid, but the larger it gets, the more framework scaffolding has to be in place so that the train stays on the rails.
So we have solution-level objectives that apply to all of the vendors across all of the contracts that pertain to this particular business unit of the federal government. And this is all USDA, so none of it is classified. It’s all agricultural stuff. Some of it supports farm bill initiatives, some of it supports conservation resource initiatives, so water conservation, land-use conservation, and things like that. So a lot of data, a lot of GIS-type data mapping runs through these systems. So having the solution level set objectives sort of gives, lack of a better term, we’ll call that the stick portion of the enticement. And then on the carrot side, the team has to go out and meet with all of these other development teams as they come up and try to integrate with the systems as service and get them to understand why we’re doing things the way that we’re doing them. And then in that process, we’ve been to do some learning from their systems people, their teams, because these companies all have their own DevOps departments who all have their own practices and their own tools and their own systems. And so we’ve been able to sort of listen to their approach and maybe pick up on things that help us improve our services as well. So sort of a two-pronged approach at the scale that we’re running at.
When to Leave Legacy Alone
Ryan: Yeah, that makes sense. So lifetimes ago, I was a civilian employee on a Naval base. I mean, I was a lifeguard, but I still had to deal with Naval policies and procedures and the Bureau of Personnel manual that was about six inches thick which lived in our document cupboard. Correct me if I’m wrong, but dealing with government systems and federal systems and that sort of things, they tend to be a lot slower to change or, to be generous, a lot more methodical in their change. So to kind of flip things around, have you ever looked at a legacy system or a legacy process, and at first glance, it was like, 鈥淥h wow, we need to update this? Why is it being done this way?鈥� But when you really dug into it, how they’re doing it’s really the best way to do it or maybe even the only way to do it for that given system or process. First, how do you identify that, and then how do you come to terms with that?
Matt: Yeah, that’s a really great question. I think the closest example I have right now is another project that we’ve been working on for the last few months of a very, very legacy application that needs a lot of updating in a lot of ways. But one of the team’s big objectives was to get it to where our pipeline could pick up code changes and deploy that application, but they didn’t have it, such that the build process and the deploy process was separated from the code itself. All of their build scripts were internal to the application. And when we started looking at it, it was amazingly elegant in its own way that they had come up with this 15 years ago, 17 years ago, and built this thing. So it all starts with this very elaborate Perl script that then called .NET application.
Typically, there is a solution file that then goes out and calls all of the different projects, or maybe you build it just at the project level, but in either case, there’s sort of a hierarchy of things in all run-through projects. Well, they didn’t build it this way. They built their project files for each of the executable units was an individual Ant script, an XML build script, that was in place of the project file. And it was called as a project, but it was an Ant script. So we couldn’t just use typical MSBuild tools, for example, to build each of those projects, compile them, and then just reassemble them in a package however we wanted. We had to go through the front door that they built, which was this massive Perl script. And so it was one that, once we realized that we looked at it and said, wow, A, this is really elegant, this is really, really well done, but at the same time, it’s not possible to reproduce this in the mechanism that you want to have it done.
So we have to meet in the middle somewhere and make some changes. And so, fortunately it’s been one that we’ve been able to collaborate really closely with the developers on and untangle this process. Our knowledge of build processes and their knowledge of the application code put together has been a powerful collaboration. And one of my teammates has put in an enormous amount of time on that project getting it to work. That’s I think a great example of that. We could just leave it alone and we could do it their way and we could stand it up on a server and have it self-install on that host, but I think the two things that come into play, one, the solution level objectives for the organization that we just talked about sort of overshadow that. And then, two, the ability to move faster, the ability to get more velocity out of the developer’s work time for that application, and bring more value to the customer is more important than that legacy process. And so they’re putting in the extra time to convert it, to untangle that, to separate the build process and the deploy process from the application code itself.
Ryan: Yeah. That’s some good advice, so even if there’s something there that has to stay there. So the one thing that comes to my mind are these mainframes in the financial sector that are still running on Fortran or COBOL.
Matt: Yeah.
Ryan: Those aren’t going to be changing anytime soon.
Matt: NASA has a bunch of those, too.
Ryan: Exactly. Even if the system is there and is not going to change, you can change how you interact with it.
Matt: Exactly.
Ryan: You can change the policies around that system to incorporate it into more modern practices in a novel way.
Matt: Right. Yeah, a good transitional state that I think represents that really well that we’ve touched on over the last year is developers themselves had always been in the habit of, even when Jenkins started deploying code for them to a server, they would be in the habit of using remote desktop to go onto the Windows web server and poke around, look at logs or update a config or whatever before they’d make a change in source. And there was an early initiative to say, all right, we want to stop using remote desktop to do any kind of post-deploy fiddling. If you missed something in a config, if you messed up a database connection, or you have a web config that’s not working correctly, you don’t go onto the server and fix it because, A, you can forget to go fix it in source code, which means the next time it deploys it’s still broken.
Or, two, you get in this vicious habit of going to validate your changes in a slow manner before you commit them to source code and then redeploy them when you could have made three or four changes, even if you messed it up, three or four changes, committed, deployed, and validated it in the time that it took you to remote desktop onto the cluster of hosts that you would have had to update to validate your change. And getting that mindset to change to the new process of commit to source and deploy because it’s cheap, that the pipeline can move the code from the repository to a deployed server in mere minutes, sometimes seconds, whereas having to remote desktop and fiddle with config can take tens of minutes to an hour to validate depending on the size of the application and the host cluster. So there’s a process there that’s been very, very slow to creep in, but people are finally realizing the value in that. And they recognize that they can do that stuff quickly and becoming more accurate.
Moving Away From Legacy IT: Trends
Ryan: Yeah. And I’m glad that you’ve mentioned remote desktop. I’ve done it a few times. It’s a really interesting trend that I’ve noticed. It was something that I noticed when changing from my last job to my current job. And that is, in my last job, I actually relied heavily on remote desktop for support purposes and that sort of thing. And here at 黑料海角91入口 I’ve used a remote desktop solution once maybe two years ago. There’s two parts of it. First off, I have better tools available to me. I can remotely pull logs or push a configuration or a policy to a system without having to remote into it. But also, as far as the practice goes, the times when I needed to see what’s on somebody’s system, I found it more useful to jump on a video conference with somebody, have them do a screen share, and walk them through some stuff.
So first off, it’s more engaging for the other person, for my customer to see what’s going on and it helps you build rapport and everything like that, but also it gives them kind of a sense of empowerment that after they’re done, they fixed that thing. I may have told them what to do, they’re the ones who fixed it and I’ve found that to be invaluable. And honestly, I don’t miss any kind of remote desktop. If I never have to do a team viewer session ever again, I’ll be more than happy.
Matt: Yeah. For sure. Yeah. That’s a really good point. In my last job working at the school, this is how I got introduced to 黑料海角91入口 because I came into a school that was hardcore, on-prem active directory, Windows only. And I came from more of a Mac background and I was more comfortable with the operating systems. I knew my way around a bunch of things. And I knew that I could much more easily script for a Unix-type operating system at that time than a Windows system, not that you can’t script for Windows. There’s a lot of really great tools out there for that, but I knew that a path forward there was to get to where we weren’t dependent on the domain controller, the mothership being local to the systems.
I wanted people to be able to take their laptops home, staff to take a laptop home, do work, and if something needed to be updated or they needed to authenticate something, we could authenticate it no matter where they were. And so I spent, I don’t know, probably three years, about every six months I would get really frustrated and I would jump on to Google and I would look for, how could I integrate – this was my starting premise – how could I integrate our Google Apps domain, because all of our email system was Google, with Active Directory systems management? And at first, there wasn’t anything. And then there were some expensive solutions that still required on-prem agents. And then, in the third year of this search, there was this new hit on my searches, this company called 黑料海角91入口.
I’m like, what’s this? And I click on it and sure enough, it’s a company in Boulder. And so I called right up and I said, tell me more. And they said, well, hey, you should set up a free account and just try it for yourself. You get 10 users and as many systems as you want to put on there, really. And this really sounds like a plug for 黑料海角91入口, but I really, really was excited about this product. And it allowed me very quickly to integrate any identity system that we had. At the time we had some SAML applications that used some open authentication methods. So we could integrate with LDAP. We could integrate with Okta. We could integrate with Google Directory. We could integrate with MS365. We could bring it all in and wrap it into one identity so everybody had one login and this was blowing people’s minds. But the big thing, back to the remote desktop point, was people needed software installed on a weekend on their system. They weren’t at the school. I wasn’t at the school. I couldn’t get to their system.
We could just pass CLI commands through the 黑料海角91入口 agent and install or update software. They get a Java update that’s blocking some applications from running correctly, well, we can just run that. We can trigger it on a schedule or a webhook. And this really freed up, not just the IT staff from having to be tied to a location, to go to a person’s deskside and physically touch their computer, or do a remote desktop session, but it allowed us to do this stuff wherever they were as long as we both had an internet connection. And it also allowed us to do it across any major platform. We could do it on Mac. We could do it on Linux. We could do it on Windows. It was universal. And that was very, very powerful to transforming a very rigid, structured system. Not a bad system, but very legacy in that people didn’t work that way anymore. People worked from everywhere. People needed to work from home. People needed to work in the office. Wherever they were, we needed to be able to provide the same level of service.
Ryan: Yeah. Well, I appreciate the plug. I may be a little bit biased, but I also like the 黑料海角91入口 platform. But I think that brings up a very good point that, beyond 黑料海角91入口, that no matter what you’re trying to do, somebody else has tried to do that in the past and there is very likely a solution out there. Yes, you’re going to have budgetary constraints and everything like that, but I think one of the important aspects of dealing with legacy systems and processes is a willingness to keep your eyes open. To go out there periodically and say, what’s out there now and what can help me with this because I had the very similar experience. As you, that’s how I learned about 黑料海角91入口. I found 黑料海角91入口 in a very similar situation for you. At the time, it wasn’t as robust as it is now and I was looking for a very niche use case, so I didn’t go with it. So instead of acquiring the product, I went and worked for the company.
Matt: There you go.
Combating Risk Aversion
Ryan: But the point is that you have to be willing to not only go out there and find new solutions but be willing to get over that bit of hesitation or fear and be willing to implement these solutions. I know that dealing with Active Directory, in particular, and a lot of the AD admins that I know, and it’s like, it works, it’s not on fire, if it ain’t broke don’t fix it. That whole mentality. I mean, setting up the AD environment is expensive, it’s time-consuming, and everything like that. So once you have it set up you don’t necessarily want to mess with it. You go into maintenance mode and you want to do that.
I think at times, instances like that, getting into that sort of mentality of, “it’s working right now so I don’t want to touch it,” ends up being counterproductive. And it’s not just an AD thing. It could be with any type of system. I mentioned before, the phone system running on Windows 95 or even a locally hosted PBX, at what point should you just be like, this is inefficient. There are better ways out there of doing this. We’ll rip off the Band-Aid. It’ll be worth it, in the long run, to rip it out and replace it with something more modern. And that’s scary because if you do that and it doesn’t work, then that comes down to you.
And so I think that creates a lot of hesitation. That makes people nervous to do that sort of thing but I feel that as long as you can communicate why you’re wanting to do that and the potential benefits of making a big systemic change, you should be fine. I mean, granted, it’s going to fall down to company culture and that sort of thing. I know here I’ve been like, “Okay, we’re going to go do this,” and it ends up not working out and we have to change direction, stuff like that. Essentially, you have to be willing to fail because you’ll learn from those failures. And then that allows you to be more successful in the future.
Matt: Yeah. I think there’s another piece to making change and making it more palatable to the risk-averse. We talked about making big changes as being risky and sometimes they are and sometimes you have to make a big change. Sometimes you’ve got to rip out a big piece of your systems and replace them with new ones. But oftentimes it can be a lot easier if you are able to take a slower iterative approach to transition and sort of boil the frog slowly, to use the phrase, and get people used to incremental change over time, rather than saying, “All right, we’re going to culture shock you all and rip out Active Directory and drop in some other management system.” Or, “We’re going to rip out our phone system that you guys have used for 25 years and we’re going to put in a cloud-hosted VoIP system,” or whatever. Those types of changes are sometimes very difficult for teams to transition on, even if they have a good understanding of why you’re doing it.
But if there’s a way to transition slowly and make incremental change that people can realize… you mentioned the Slack approach to tickets. There’s multiple entry points to the ticketing system, and people can start to use the methods that they’re comfortable with, whether you can email, you can send a Slack message, you can log into this portal, whatever methods are there. If you give people options so they can transition at their own rate while still using and leveraging the power of the tools that you’ve stood up, sort of the black box tools that they don’t need to worry about, they put something in the machine and results come out, that’s really what a lot of people want to see. So if we can make an incremental change and move the needle slowly for people so that they are comfortable with the risk that they have to take, and then if we can black box a lot of the behind-the-scenes heavy lifting, I think those are two big points that we can win on.
Ryan: Yeah, I agree. I’ve been in the situation of, I’m going to go back to the phone system because I may be a little bit scarred from dealing with some phone systems in the past, but I’ve done the migration from a traditional PBX and trunk line and that sort of stuff to a VoIP system. And you’re right, intermediate steps. We went going from a traditional self-hosted PBX with a trunk line coming in, everything like that, to a self-hosted PBX that has a VoIP trunk coming in and changing that part of the system. And the people who are using the phones, they don’t know the difference.
It’s all the same for them. And then the next step is moving the PBX from self-hosted up onto the cloud and stuff like that. And again, still keeping the same phones. There may be a little bit of change with how you may check your voicemail and that sort of stuff, but yeah, doing these little incremental changes and making it so even if there’s massive differences on the backend, like you said, just black box it, the person who’s using the service doesn’t know the difference.
Matt: Right. And they don’t care necessarily.
Ryan: Exactly. And that’s a very good point. They’ll say, as long as it keeps working and it works how they expect it to work.
Matt: Exactly. Yeah.
Ryan: They don’t care what’s under the hood.
Matt: Right. Yeah. If they can get from point A to point B without being uncomfortable.
Ryan: Exactly. And I think that’s where when you’re assessing new tools and that sort of thing, demos and test accounts and that sort of stuff, are super valuable and bring in one of those stakeholders. If you can get the most ornery, stuck-in-their-ways end-user possible to join you to demo this new system. I’ve experienced it before where they’re like, “What’s the difference? Why are we even making this change? It’s all the same to me.” That’s the magic moment. If you can get that person to say that, you know you’re moving in the right direction. And the buy-in from the people who are going to be using the systems is going to be easy.
Collaborative Improvement
Matt: Yeah. That’s great. And I think sort of speaks to another point along that same line in that getting people to recognize and encouraging people, whether it’s direct teammates or whether it’s that cantankerous person who you want their buy-in so that they’re willing to use the changeset that you’re putting in their way is to get them to start thinking and encourage them to think about ways that their life could be easier with any system, whether it’s a technology system or a process or something. When I was young, I was in high school and I worked for Fort Collins nursery here in Fort Collins, Colorado, and my job every summer was plant propagation. And so we would go out and take small cuttings off of the end of hundreds of varieties of plants and we would dip them in a little rooting hormone, and we would stick them in the soil and they would root and then they would eventually grow up to be sellable plants.
Well, the guy who I worked for, first day on the job, every summer, the whole team would come in and we’d sit down and he would lay it out there. He’s like, 鈥淚f you can think of a way to do this faster with better results, I will pay you a bonus.鈥� He’s like, 鈥淚 don’t think you’re going to find one, but if you do, you let me know, we’ll test it and if it works, and if it proves itself, you’ll get benefit for that.鈥�
And I think that stuck with me over the years. And I haven’t always been so mindful of it, but I think if you can find those people who are, A, either the ones who really want to see change, or the ones who really don’t want to see change – I think those are the two most powerful assets you can have if you can balance them together – and say, “All right, you come up with a reason to stay the way we’re doing it that is beneficial. Or you come up with reasons and methods or processes that we can change to improve things then we’ll definitely recognize that and we’ll move it.” And gathering that input from the team or the customer or whoever your stakeholders are is both really powerful for change managers, but also really powerful and empowering to those customers to come and speak to those providers and get collaborative input and buy-in from both ends.
Ryan: Yeah. I completely agree. We’ve implemented changes based on constructive criticism that we’ve received and we’ve made it a point when we do somebody’s first-day IT orientation when they start. First off, the fact that we do an IT-specific orientation for all new hires, I feel is a good way to start building that rapport with the end-users.
Matt: It seems like a no-brainer for a tech company to do an IT orientation, right? Like you’d easily look right past that.
Ryan: Yeah. Granted, this is not the first tech company that I’ve worked for. I interned at Google and your orientation was days long, but it wasn’t necessarily IT, tech-specific, it was general company stuff and then you split off and you did more departmental role-specific training beyond that. But here, everybody, no matter the role, like all engineers, sales, marketing people, they’re all in this same IT orientation. And then we may do specialized stuff in future sessions for specific roles. But during this, we make it known that, first off, we want to be the best. We want to be the best IT team that you’ve ever worked with, period.
And the approach that we take is that in order to do this, we are open to criticism, comments, suggestions. If you worked with another team that you loved how they did this one procedure, let us know about this and we’ll assess it. And it comes down to, if we have a policy in place that you don’t like, ask us, bring it up to us and tell us. If it’s something that we can’t change, we’ll express it. We’ll say why we’re doing it, which leads me onto kind of the roundabout way that I was trying to make this point, is that with our policies that are in place, I feel that it’s super important to be able to articulate why the policy is in place. If you can’t articulate why the policy is in place, maybe the policy needs to change. Or maybe you just need to refresh yourself on why it’s there, looking back on the whole overwriting the storage device when imaging a new system. It made sense then. It doesn’t make sense now when you try to articulate it to somebody. That will bring that up to the surface and be like, 鈥淭his doesn’t make sense anymore. Let’s change.鈥� And that makes it empowering for the person who brought up the issue.
Matt: Sure. I think you bring up another good thought in my mind about this, that these policies and procedures sometimes have unintended consequences. Take your imaging of a system with, this starts with a clean wipe. Well, you do that on an SSD and that can be degrading to the disc, right? So a policy that had the best of intentions, largely security-related, now can actually cause damage to a system potentially if that policy doesn’t change. The same thing happens if you’re going to implement a new policy, whatever the policy might be. This is where collaboration and getting a few more cooks in the kitchen really can make a difference, is that one person says, oh, it’d be a great idea if we set a policy on all of our users’ systems that they can no longer do X. Well, did you consider the fact that if they can’t do X, that now you might have to do X on their behalf all the time? And if we don’t have conversations with our stakeholders or with other people on the teams, we might introduce adverse consequences to those decisions in those policies and procedures.
Ryan: Yeah. That’s very true. I’ve had good ideas come back and bite me in the past. I think any kind of decision maker has had that happen. So I think the important part of that is to, again, don’t let pride get in your way. Admit that you’re wrong. You’ll learn from it. It’ll be like, okay, never mind.
Matt: Yep. That was a bad idea.
Ryan: That was a bad idea. And also the side effect of these practices of bringing in stakeholders, of listening to criticism, of admitting when you’re wrong and taking action to fix mistakes that you’ve made, make it so, in future decisions if you want to make future changes, I found that people are more willing to trust you, to go along with your decisions and to be like, “Okay, there’s good reason.”
There are times when I try to be as transparent as possible to my team members, to my coworkers, customers, whatever I am calling them at the moment. I try to be as transparent as possible, but there are some things where I’m privy to sensitive information that I cannot share to the broader public. And so when making these decisions and expressing why we do things the way that we do, if I normally maintain a high level of transparency and I’m frank and honest about why we’re doing things, or we made a mistake, or anything like that when you get to those points where you’re like, “We’re doing this, I can’t go into why,” you’ve already built that trust. And again, that’s a good way of being able to make these changes in the future, move away from these legacy systems. I think that’s an important part of change management.
Matt: Yeah, for sure. And I think that relationship component and the trust component is huge. That theme has come up several times in our conversation today. And I think that’s the crux of it, right? If we can be fully human in our relationships and not just present the wins, not the positives, the things that we’ve accomplished, or the things that we’re going to accomplish, but we can also turn around and say, “Yep, that was a bad idea, I’ll own that one,” and we can have a good laugh about it, that really shows people that we’re real, because really everyone has that sort of as an insecurity. Right. We’re all slightly afraid of showing the negative parts of our ourselves and the flaws in ourselves. And if we can do that, if we can be vulnerable with our teams, with our stakeholders to the point where it’s appropriate of course, and admit the fact that, hey, we’re human too, we messed this up, and here’s what we’re going to do to make it right, I think those are really powerful for our customers.
Wrapping Up
Ryan: Well, that’s all the time that we have today. We’ve talked about a lot of good stuff here. So thanks, Matt. Again, our guest today is Matt Cramer. Thank you very much for coming on and chatting with me. It was a lot of fun.
Matt: Yeah. Thank you so much. I really enjoyed it. Good conversation.
Ryan: Thank you for tuning in to Where’s The Any Key? If you like what you heard, please feel free to subscribe. Again, my name is Ryan Bacon. I lead IT at 黑料海角91入口 where the team here is building a cloud-based directory platform that provides frictionless secure access to virtually any IT resource from trusted devices anywhere. You can learn more and even set up a free account at
About 黑料海角91入口
黑料海角91入口 is a cloud-based directory platform that helps companies migrate from legacy directory services to domainless enterprise environments that combine identity, access and device management. It鈥檚 free to try – to make remote work happen in your organization.