The Phoenix Project

"Give me a break. You're bored in your current role. You're going to become a lot less bored. Fast. And remember that there are a lot of experienced people around you

who've been on similar journeys, so don't be the idiot that fails because he didn't ask for help."

My board holds me responsible for making the best use of company resources to achieve the goals that maximize shareholder value. My primary job is to lead my management team to make that happen.

I'm surprised at the unanticipated payoffs of automating our deployment process. The developers can more quickly scale the application, and potentially few changes would be required from us.

At the same time, we wouldn't get bogged down in the processes that we didn't need to be a part of.

"I need you to help me elevate the state of the practice of how organizations manage technology. Let's face it. Life in IT is pretty shitty when it's so misunderstood and mismanaged. It becomes thankless and frustrating as people realize that they are powerless to change the outcome, like an endlessly repeating horror movie. If that's not damaging to our self-worth as human beings, I don't know what is. That's got to change,"

"In ten years, I'm certain every COO worth their salt will have come from IT. Any COO who doesn't intimately understand the IT systems that actually run the business is just an empty suit, relying on someone else to do their job."

I don't want posters about quality and security. I want improvement of our daily work showing up where it needs to be: in our daily work.

Because of our rapid progress, we decided to shrink the sprint interval to two weeks. By doing this, we could reduce our planning horizon, to make and execute decisions more frequently, as opposed to sticking to a plan made almost a month ago.

In many ways, most of these guys are my temperamental opposites. I like people who create and follow processes, people who value rigor and discipline. These guys shun process in favor of whim and whimsy.

you need to create what Humble and Farley called a deployment pipeline.

That's your entire value stream from code check-in to production.

"How Toyota solved this problem is legendary," he says. "During the 1950s, they had a hood stamping process that had a changeover time of almost three days. It required moving huge, heavy dies that weighed many tons. Like us, the setup times were so long that they needed to use large batch sizes, which prevented them from using one stamping machine to manufacture multiple different car models simultaneously. You can't make one hood for a Prius and then one hood for a Camry if it takes you three days to do the changeovers, right?

"What did they do?" he asks rhetorically. "They closely observed all the steps required to do the changeover, and then put in a series of preparations and improvements that brought the changeover time down to under ten minutes. And that, of course, is where the legendary 'single-minute exchange of die' term comes from

"In any system of work, the theoretical ideal is single-piece flow, which maximizes throughput and minimizes variance. You get there by continually reducing batch sizes.

"You're doing the exact opposite by lengthening the Phoenix release intervals and increasing the number of features in each release. You've even lost the ability to control variance from one release to the next."

the Second Way, creating constant feedback loops from IT Operations

back into Development, designing quality into the product at the earliest stages. To do that, you can't have nine-month-long releases. You need much faster feedback.

We need to be nimble, and sometimes we need to buy instead of build.

Steve has a blind spot for Sarah's shortcomings, because she has something he needs and admires, which is the ability to come up with creative strategies, regardless of whether the strategy is good or bad.

As Erik keeps reminding me, a great team performs best when they practice. Practice creates habits, and habits create mastery of any process or skill. Whether it's calisthenics, sports training, playing a musical instrument, or in my experience, the endless drilling we did in the Marines. Repetition, especially for things that require teamwork, creates trust and transparency.

When this happens now, we quickly jump on it to make sure that it doesn't happen again.

The need to continually reduce cycle times is part of the First Way. The need for amplification of feedback loops, ideally from the customer, is part of the Second Way.

"Some of the wisest auditors say that there are only three internal control objectives: to gain assurance for reliability of financial reporting, compliance with laws and regulations, and efficiency and effectiveness of operations.

William walks to the whiteboard and points at a box called "code commit." "If I could wave this magic wand, I would change this step. Instead of getting source code or compiled code from Dev through source control, I want packaged code that's ready to be deployed."

As part of the Second Way, you need to create a feedback loop that goes all the way back to the earliest parts of product definition, design, and development,"

"You'll never hit the target you're aiming at if you can fire the cannon only once every nine months. Stop thinking about Civil War era cannons. Think antiaircraft guns."

"Metaphors like oil changes help people make that connection. Preventive oil changes and vehicle maintenance policies are like preventive vendor patches and change management policies. By showing how IT risks jeopardize business performance measures, you can start making better business decisions.

But you can also count on me calling 'bullshit' if I see Patty trying to create some sort of bureaucracy that sucks out everybody's will to live.

Are we competitive?

Understanding customer needs and wants: Do we know what to build?

Product portfolio: Do we have the right products?

R&D effectiveness: Can we build it effectively?

Time to market: Can we ship it soon enough to matter?

Sales pipeline: Can we convert products to interested prospects?

Are we effective?

Customer on-time delivery: Are customers getting what we promised them?

Customer retention: Are we gaining or losing customers?

Sales forecast accuracy: Can we factor this into our sales planning process?

Steve's incredibly charismatic, and let's face it, I'm an asshole. But when people have concerns, they don't want to have their minds changed. They want someone to listen to them and help make sure Steve gets the message."

"I know. But, in my world, I'm responsible for keeping all our services up and running, and deploying new services like Phoenix. Security had to take a backseat. Trust me, I'm well aware of the risks of bad security, and I know it would be a career- ending move to have a large-scale security breach on my watch."

I shrug my shoulders. "I'm making the best possible decisions given my knowledge of the risks. I just don't think that all the

stuff you wanted me to do would have helped the business as much as all the other things on my plate.

Startled, I realize he's talking about Erik's First Way. He's talking about systems thinking, always confirming that the entire organization achieves its goal, not just one part of it.

"Wouldn't you agree, like so many things in life, that we must always start by at least gaining an understanding of the person we're dealing with? What could go wrong? I just want to learn more about his job."

"To tell the truth is an act of love. To withhold the truth is an act of hate. Or worse, apathy."

I had laughed at those words at the time, but over the years, I've realized that having people give you honest feedback is a rare gift.

Creating and prioritizing work inside a department is hard.

Managing work among departments must be at least ten times more difficult.

"You know, I did a quick poll of people we've issued laptops to. It usually takes fifteen turns to finally get them configured correctly. I'm tracking that now, and trying to drive this down to three. We're putting in checklists everywhere, especially when we do handoffs within the team. It's really making a difference. Error rates are way down."

"I was thinking the same thing. Imagine what this will do to user satisfaction if we could tell them when they make the request how long the queue is, tell them to the day when they'll get it, and actually hit the date, because we're not letting our workers multitask or get interrupted!

"Our auditors should be put on trial for incompetence! All those findings they dismissed were basic hygiene issues! We live in a goddamned cesspool of risk. I'm amazed this place doesn't just collapse under its own weight from lack of caring. I've waited for years for everything to come crashing down upon us."

"Mike Rother says that it almost doesn't matter what you improve, as long as you're improving something. Why? Because if you are not improving, entropy guarantees that you are actually getting worse, which ensures that there is no path to zero errors, zero work-related accidents, and zero loss."

Properly elevating preventive work is at the heart of programs like Total Productive Maintenance, which has been embraced by the Lean Community. TPM insists that we do whatever it takes to assure machine availability by elevating maintenance.

It's amazing how the project freeze has reduced the priority conflicts and bad multitasking. We know it's made a huge difference in productivity."

When you spend all your time firefighting, there's little time or energy left for planning. When all you do is react, there's not enough time to do the hard mental work of figuring out whether you can accept new work. So, more projects are crammed onto the plate, with fewer cycles available to each one, which means more bad multitasking, more escalations from poor code, which mean more shortcuts.

four types of IT Operations work: business projects, IT Operations projects, changes, and unplanned work.

Solving any complex business problem requires teamwork, and teamwork requires trust.

One is that IT matters. IT is not just a department that I can delegate away. IT is smack in the middle of every major company effort we have and is critical to almost every aspect of daily operations."

when I first became a plant manager, many decades ago, I worked on the assembly line for a month, just to make sure that I understood the ins and outs of daily life of everyone who worked there.

Being able to take needless work out of the system is more important than being able to put more work into the system. To do that, you need to know what matters to the achievement of the business objectives, whether it's projects, operations, strategy, compliance with laws and regulations, security, or whatever.

That's why Erik called it the most destructive type of work. It's not really work at all, like the others. The others are what you planned on doing, allegedly because you needed to do it.

Unplanned work is what prevents you from doing it. Like matter and antimatter, in the presence of unplanned work, all planned work ignites with incandescent fury, incinerating everything around it. Like Phoenix.

What's happening with Phoenix is a combination of the need to deliver needed features to market, forcing us to take shortcuts, which are causing ever-worsening deployments.

This should be your guiding principle: you win when you protect the organization without putting meaningless work into the IT system. And you win even more when you can take meaningless work out of the IT system."

I wanted to help catch bad guys. I wanted to help organizations protect themselves from people who were out to get them. It came out of a sense of duty and a desire to make the world a better place.

'Improving daily work is even more important than doing daily work.'

every work center is made up of four things: the machine, the man, the method, and the measures.

I'm pretty sure we don't do any sort of analysis of capacity and demand before we accept work. Which means we're always scrambling, having to take shortcuts, which means more fragile applications in production. Which means more unplanned work and firefighting in the future. So, around and around we go.

I'm just stating a well-known fact. When deployments don't go as planned, whether the plan was written by your group or mine, it affects everybody else.

I've seen such a difference in how the entire IT organization works. This is an organization that has resisted adopting any sort of process and had real problems with trust between departments. It's amazing to see, and I give most of the credit to you."

"I know that the last couple of weeks have been harrowing. I now realize just how much responsibility I bear for all of this. Not just for the Phoenix disaster, but everything leading up to the audit issues, the customer invoicing and inventory failures over the last couple of days, and the trouble we're having with the auditors."

"Remember, outcomes are what matter—not the

process, not controls, or, for that matter, what work you complete."

"Precisely!" I hear Erik say. "You even used the term I like most for it: unplanned work. Firefighting is vividly descriptive, but 'unplanned work' is even better. It might even be better to call it 'anti-work,' since it further highlights its destructive and avoidable nature.

"Unlike the other categories of work, unplanned work is recovery work, which almost always takes you away from your goals. That's why it's so important to know where your unplanned work is coming from."

A fellow NCO in the Marines once told me that his priorities were the following: provider, parent, spouse, and change agent. In that order.

"It's like the free puppy," I continue. "It's not the upfront capital that kills you, it's the operations and maintenance on the back end."

"Second, I'm done playing Russian roulette with IT. Phoenix just shows me that IT is a competency that we may not be able to develop here. Maybe it's not in our DNA. I've given Dick the green light to investigate outsourcing all of IT and asked him to select a vendor in ninety days."

Outsourcing all of IT. Holy shit.

"We've done a terrible disservice to our customers. They're the people who need to fix their cars to get to work. They're fathers working on projects with their kids. We've also screwed some of our best suppliers and clients.

"Bad news. In short, it's all over Twitter that the Phoenix website is leaking customer credit card numbers. They're even posting screenshots. Apparently, when you empty your shopping cart, the session crashes and displays the credit card number of the last successful order."

"It's crazy what programmers, and even managers like me, have to learn every couple of years.

Sometimes it's a totally new database technology, a new programming or project management method, or a new technology delivery model, like cloud computing.

"Just how many times can you throw out everything you know to keep up with the latest new-fangled trend? I look in the mirror

every once in awhile, asking myself, 'Will this be the year that I give up? Will I spend the rest of my career doing COBOL maintenance or become just another has-been middle manager?'"

IT is in the way of every major initiative.

I hear her sigh disapprovingly. "You've been there for over ten years and you've never worked these kind of hours. I'm really not sure I like this promotion."

"You and me both, honey…" I say.

"Processes are supposed to protect people. We need to figure out how to protect Brent,"

Efforts like this must start and be continually maintained from the top.

I arrive and immediately see Kirsten Fingle, who runs the Project Management Office sitting at the head of the table. I am a big fan of hers. She is organized, levelheaded, and a stickler for accountability. When she first joined the company five years ago, she brought a whole new level of professionalism to our organization.

It's amazing to me how handoffs between Development and IT Operations always get screwed up. But given the perpetual tribal warfare between the two groups, maybe I shouldn't be surprised.

Three Ways," he says. "The First Way helps us understand how to create fast flow of work as it moves from Development into IT Operations, because that's what's between the business and the customer. The Second Way shows us how to shorten and amplify feedback loops, so we can fix quality at the source and avoid rework. And the Third Way shows us how to create a culture that simultaneously fosters experimentation, learning from failure, and understanding that repetition and practice are the prerequisites to mastery."

"Eliyahu M. Goldratt, who created the Theory of Constraints, showed us how any improvements made anywhere besides the bottleneck are an illusion. Astonishing, but true! Any improvement made after the bottleneck is useless, because it will always remain starved, waiting for work from the bottleneck. And any improvements made before the bottleneck merely results in more inventory piling up at the bottleneck."

You've probably heard of them: the Theory of Constraints, Lean production or the Toyota Production System, and Total Quality Management. Although each movement started in different places, they all agree on one thing: WIP is the silent killer.

"You probably don't even see when work is committed to your organization. And if you can't see it, you can't manage it—let alone organize it, sequence it, and have any assurance that your resources can complete it."

"Practically every executive in the company is guilty of going directly to their favorite IT person, either asking a favor or pressuring them to get something done."

, "Settle down, everyone. We all want to do what's right for the company. The trick is figuring out what we have time to do and what systems can actually be patched."

He holds up one page. "'Issue 127. Insecure Windows operating system MAX_SYN_COOKIE setting'? Is this a joke? In case you haven't heard, we've got a real business to run. Sorry if that interferes with this full-time audit employment racket you've got going on here."

I always liked that phrase in Saving Private Ryan: "There's a chain of command: gripes go up, not down."

I've seen this movie before. The plot is simple: First, you take an urgent date- driven project, where the shipment date cannot be delayed because of external commitments made to Wall Street or customers. Then you add a bunch of developers who use up all the time in the schedule, leaving no time for testing or operations deployment. And because no one is willing to slip the deployment date, everyone after Development has to take outrageous and unacceptable shortcuts to hit the date.

The results are never pretty. Usually, the software product is so unstable and unusable that even the people who were screaming for it end up saying that it's not worth shipping. And it's always IT Operations who still has to stay up all night, rebooting servers hourly to compensate for crappy code, doing whatever heroics are required to hide from the rest of the world just how bad things really are.

To help with Phoenix, his team has grown by fifty people in the last two years, many through offshore development shops. Chris is constantly asked to deliver more features and do it in less time, with less money.

We've got to get this nailed down— grab everyone involved, whether they're in Dev or Ops, and lock them in a room until they come up with a written specification.

In my old group, we were always disciplined about doing changes. No one made changes without telling everyone else, and we'd bend over backward to make sure our changes wouldn't screw someone else up.

I'm not used to flying this blind.

exploit to pillage our entire organization and steal all our code, intellectual property, credit card numbers, and pictures of our loved ones. These are potentially valid risks, but I often can't connect the dots between their shrill, hysterical, and self- righteous demands and actually improving the defensibility of our environment.

"Yeah, I was helping someone who was having some connectivity issues with the plants. I'm pretty sure he was a developer maintaining the timekeeping app. He was installing some security application that John needed to get up and running this week. I think his name was Max. I still have his contact information around here somewhere… He said he was going on vacation today, which is why the work was so urgent."

Show me a developer who isn't crashing production systems, and I'll show you one who can't fog a mirror. Or more likely, is on vacation.

"I need you to keep all the things that are supposed to be up, well, up. I need someone reliable, who isn't afraid to tell me bad news. Above all, I need someone I can trust to do the right thing. That integration project had many challenges, but you always kept a cool head. You've built a reputation as someone who is dependable, pragmatic, and willing to say what you really think."

Despite all our hard work and late nights, the competition keeps leapfrogging us. When our Marketing people hear this ad, they'll go ballistic. Because they're likely art or music majors, not people with a technology background, they'll publicly promise the impossible and IT will have to figure out how to deliver.

The First Way is all about controlling the flow of work from Development to IT Operations.

"When we're out of the woods," I say, "we've got to figure out how to prevent this from happening again. There should be

absolutely no way that the Dev and QA environments don't match the production environment.

For years, Steve searched for a strategy person to be his right- hand man. He went through quite a few people, even setting a couple executives against each other in this awful, drawn out competition. Pretty Machiavellian

First Way: We're curbing the handoffs of defects to downstream work centers, managing the flow of work, setting the tempo by our constraints

People are getting satisfaction out of fixing things. I'm hearing that people are feeling happier and more upbeat, because they can actually do their jobs.

Sarah is the SVP of Retail Operations, and she also works for Steve. She has an uncanny knack for blaming other people for her screwups, especially IT people. For years, she's been able to escape any sort of real accountability.

Sarah is the SVP of Retail Operations, and she also works for Steve. She has an uncanny knack for blaming other people for her screwups, especially IT people. For years, she's been able to escape any sort of real accountability.

"You want the IT systems to be reliable and available, and for the business to be able to depend upon them. You want disruptions to normal operations kept to an absolute minimum so that the business can focus on getting Phoenix done."

They also knew that until code is in production, no value is actually being generated, because it's merely WIP stuck in the system.

"As part of the First Way, you must gain a true understanding of the business system that IT operates in. W. Edwards Deming called this 'appreciation for the system.'

But unlike before, our challenges feel within our ability to understand and conquer. Our goals finally seem achievable. I no longer feel like I am always on my heels, with more and more people piling on, trying to push me over.

And like when Mr. Sphere told everyone in Flatland, you must leave the realm of IT to discover where the business relies on IT to achieve its goals."

"This is bullshit! You're asking me to prove that we didn't do anything. How the hell do you prove a negative? Besides, I'm guessing the problem is a bad firewall change. Most of the outages in the last couple of weeks were caused by one of them."

Patty says adamantly, "It should be me. I've been trying to keep John's people in line for years. They never follow our process, and it always causes problems. I'd love to see Steve and Dick rake him over the coals for pulling a stunt like this."

He just made up an arbitrary date to go into production, with complete disregard for all the things we need to do before deployment.

"You know what your problem is?" Erik says, pointing a finger at John. "You never see the end-to-end business process, so I guarantee you that many of the controls you want to put in aren't even necessary."

When he mentioned "work," he seemed to mean it at an organizational level not at the level of an individual contributor or manager.

"Your job as VP of IT Operations is to ensure the fast, predictable, and uninterrupted flow of planned work that delivers value to the business while minimizing the impact and disruption of unplanned work, so you can provide stable, predictable, and secure IT service."

"I suppose to him, it sounds like a reasonable way to pick which job to perform. He's keeping the first station busy, and it's similar to first-in, first-out scheduling. But of course, now everyone knows that you don't release work based on the availability of the first station. Instead, it should be based on the tempo of how quickly the bottleneck resource can consume the work."

In hindsight, we now know that WIP is one of the root causes for chronic due-date problems, quality issues, and expediters having to rejuggle priorities every day. It's amazing that this business didn't go under as a result."

"a 'change' is any activity that is physical, logical, or virtual to applications, databases, operating systems, networks, or hardware that could impact services being delivered."

Wes says, "Half quit within a year, and I'm not getting anywhere near the productivity I need from the ones who stayed. Although I don't have data to prove it, I'm guessing Brent is even more behind than ever. He complains that he had to spend a bunch of time training and helping the new guys, and is now stretched thinner than ever. And he's still in the middle of everything."

It's the never-ending hamster wheel of pain: Information Security fills up people's inboxes with never-ending lists of critical security remediation work, quarter after quarter.

"That's what I've been trying to tell you. John rarely goes through our change process. Nor do most people, for that matter. It's like the Wild West out here. We're mostly shooting from the hip." She says defensively. "We need more process around here and better support from the top, including IT process tooling and training. Everyone thinks that the real way to get work done is to just do it. That makes my job nearly impossible."

I suddenly regret my choice of words. This isn't a witch hunt, and I'm not looking for retribution. We still need a timeline of all relevant events leading up to the failure.

"Luke and Damon are gone?" John sounds surprised. "I never thought that Steve would actually fire both of them over a compliance audit finding. But who knows? Maybe things are finally starting to change around here. Let this be a lesson to you, Bill. You Operations people can't keep dragging your feet on security issues anymore! Just some friendly advice…

Information Security is always flashing their badges at people and making urgent demands, regardless of the consequences to the rest of the organization, which is why we don't invite them to many meetings. The best way to make sure something doesn't get done is to have them in the room.

They're always coming up with a million reasons why anything we do will create a security hole that alien space-hackers will

CIO stands for "Career Is Over."