Configuration Management for Earthlings

Configuration Management (CM) or change management is one of those mom-and-apple-pie things that no reasonable person will say much against. Of course we have to manage our resources in a documented fashion. But the way it's described in the texts and by consultants makes it seem like an unattainable dream and a maintenance nightmare all at once. They'll tell you you have to document your assets to the nth degree, and surround the entire process with elaborate "artifacts" (their fancy name for documents). Project management is similar. I once had an instructor tell us in class that good project management takes about 80% of all the effort involved in a project. While that may be good news for project management consultants, I think it's insane. No one can afford that, so no one does it, and still produce what they intended to. But while we will have projects we must manage, we can evade managing our configuration. It's ugly, but it happens. I know because I work in a data processing division with a $280M yearly budget that has not done configuration management at a high level.

Oh, we control program transmittals. But that is one of the few areas in which close tabs are kept by management of what we have out there and how it gets changed, at a level higher than the site. The technical guys are pretty good about controlling what they're in charge of and keeping it running. But our executive has no repeatable method for making decisions about what mainframes to buy or eliminate, or what sort of contract to sign this year (if any) for disk upgrades or new software. I have it on good authority that she recently lamented that she was "tired of making all these decisions by myself". Well, so are a lot of other people. She needs help. No one is knowledgeable enough make those decisions by herself (and she is definitely not an exception).

In a couple weeks, I'll know if my suggestions for jumping into CM at her level are going to be favorably received. I doubt it, but I can hope that I'm wrong--or that she is replaced soon. The charter she approved for the configuration control board had her making all the decisions, and she hasn't even implemented that. The budget is always cloaked in secrecy, and no one knows for sure what our priorities are or how much is available. There are always questionable contracts signed in late September (at the end of the fiscal year) to spend money no one else knew was there.

I'll post more on this. But for now, I have two fears: that our executive will sabotage any effort to do true CM bcause she is jealous of her unchecked power; and that the CM nazis in my agency will use the perfect to bludgeon the good into a bloody pulp, forcing us to spend all our time writing "Requirements Traceability Matrices" and the like before we can even start. You should see what they require for an "audit" or "assessment" of or CM process. Well, here's part of what they want to see:

Change Forms / Documents
Requirements Specifications
RTM
Engineering Drawings
Interface Control Documents
Schematics
CS Structure Chart
Other Technical Documentation
Deployment Management Plan
Transition to Support Plan
Computer Operations Handbook
System Test Plan
System Test Descriptions
System Test Report
Product Baseline Documentation
SW VDD and/or HW Release Record
Post Implementation Review Report

You get the idea. I'm not saying this stuff isn't all good to have. We do have some of it. I'm saying that when we're in a situation in which there is no CM process for spending hundreds of millions of dollars, it isn't helpful to demand the impossible of any attempt to get a handle on that situation. I'm for just starting, and worrying later about getting all those documents written. Even if many of them never do get produced, we'll be a lot better off than we are now.

I'll let you know how it goes.

Sorry for the Hiatus

It's mostly laziness that I haven't kept this blog up. But I've also landed in a job here that has some chance of allowing me to change the way part of IRS does business. I'll be crafting a configuration management structure that may (if the execs allow it) improve the decisionmaking around here ("here" being the organization that runs the mainframes and servers our agency uses). Hope springs eternal. I'll resist the temptation to bitch about the things I don't like about how we do things unless they shed light on government-wide problems as I see it.

Federal Government Contracting

This is one of my biggest complaints about the Federal govt. Today's Washington Post has an important article that has implications far beyond the FAA.

FAA Orders a Closer Look at Contracts

By Sara Kehaulani Goo

Washington Post Staff Writer
Sunday, August 14, 2005; Page A01

Debra Srite got an odd phone call a month after starting her new job at the Federal Aviation Administration last year.

A contractor told her that he needed several thousand dollars more from the agency because he had run through nearly all the money allowed under his company's program. When Srite, a by-the-book contracting officer who cites the United States Code from memory, pressed him to provide a reason for the new money, she said he wouldn't cooperate.

"It was like he was in charge and he was surprised I asked for that verification," she said.

The call prompted Srite to dig into the FAA's records on the project. What she and a co-worker found out about the $16 million contract raised concerns about how the FAA handles $1.3 billion in such programs with small businesses...

Among the red flags Srite says she found in the $16 million contract were invoices for an unapproved trip to Las Vegas and an unexplained lease for a Porsche Boxster. She also learned that Crown had hired the wife of a top FAA official and that the company later was ordered by that official to keep his wife on staff after the contract ended. A computer software system the firm was supposed to develop for the FAA failed many government tests and was never used, according to FAA documents...

My agency, the IRS, is not immune from contracting problems. I have referred some practices to our Inspector General (TIGTA), which resulted in a scathing report but no change in actual contracting procedures. I have even seen the wife of an executive being a high-profile employee of a contractor her husband steered a training contract to, a very similar situation to the one involving Cirillo at the FAA. (This occurred about 12 years ago, as I recall. The executive in question was canned for that and other sharp practices.) But the big waste is not due to payoffs to govt decisionmakers. Rather, it is due to perverse incentives and laziness. It's just easier to avoid competition and to give the contractors what they want, because that's what Congress seems to want. Add to this the natural desire for the private sector to make as much money as possible for the least expense and effort possible, and the push by the last several administrations to reduce the Federal workforce and hire contractors to do govt functions, and you get the situation we have today. From that same article:

Experts in contracting law say projects like Crown's are meant to save the government money but instead may add to costs. So-called support services contracts allow the FAA to award initial contracts to a short list of pre-qualified small companies and then add tasks amounting to as much as several million dollars without bidding. Such awards are often open-ended or loosely defined jobs to provide technical expertise, consultation or additional labor hours on programs ranging from developing a new software system to running a computer help desk.

Sometimes, such contracts do not include firm deadlines and in most cases they lack specific performance standards. The contractor is paid based on whether it supplied the required hours of service. In some cases, an FAA employee will retire from the government and soon after begin similar work for a contractor for more money, raising the questions of whether the government is really saving money by using the contractor.

Bingo. When I first got involved in contracting (the 1980s), the Brooks Act and the GSBCA were in full force. The IRS still made some horrific contract awards. (Purchasing Sperry-Univac computers in 1981 for our case processing was our biggest IT mistake ever, and the taxpayer is still paying for it. That original contract was thrown to the winner for what appeared to be a blatant payoff in the form of a job with them later. I knew the technical people on the award team, and they told me how they were ordered to rerun tests until the favored vendor could pass them. Additional unethical behavior occurred in the months after award as well, tripling the price.) But we've kept Unisys for reasons having to do with lack of vision and lack of will more than lack of integrity.) But we did have someone looking over our shoulder, and we gave competition a shot most of the time. This meant that requirements were written and distributed in an RFP, and we had a formal process for selecting the awardee on the basis of publicly accessible "best value" criteria. Today, all that is considered quaint. IRS has a number of "blanket purchase agreements" (BPAs) or "requirements contracts" (e.g. one with Unisys, another with Presidio) that our Procurement organization simply adds products to at whim. There is no competition, no written requirement, no RFP, no justification for other than full and open competition (JOFOC), and no documentation of any sort. Those contracts were originally rewarded for specific needs, but the Contracting Officers (COs) on them have interpreted their warrant to allow them to transform them into a one-stop shop for whatever any executive takes it into his head to fund. The vendors love it and make it easy for the CO. All the vendors have jumped on board and no one wants to upset the applecart by complaining too loudly about losing the opportunity to bid on a requirement. We now buy hardware, software, and services this way.

This lack of discipline has cost the taxpayer many millions of dollars. I have specific examples, many of which I have shared with TIGTA, but I'll refrain from citing them for legal reasons. None of the recent ones, to my knowledge, involves the sort of bribery shown in the FAA case. But it's still wrong, isn't it? COs are not doing their jobs, and executives are practicing gross incompetence. It's gotten to the point that nearly everyone just throws up their hands and claims that there's nothing anyone can do about it. Actually, it's been at that point for about 15 years. But it's bullshit. A President who cared could turn it around, but we haven't had one for a very long time. Instead, we have a destructive alliance between a log-rolling Congress, self-interested executive branch executives, lazy employees, and greedy contractors. No one is effectively advocating for good government or for the taxpayer. (The taxpayers have always been lazy about things like this anyway. I don't blame them too much--it's complicated. Even the Halliburton outrage seems to sail right over their heads.)

I'll return to this in future posts. It's big; we're talking about not only the wasted money, but the lost opportunity for doing things right.

What I'll Be Doing Here

I'm not entirely sure, but my intention is to sound off on broadly work-related issues. I'm a Federal bureaucrat, and there's a lot going on in Washington that I don't like. It will be a little bit political, but Presidents from both parties have been committing some gross executive branch malpractice, so I'm not going to just bash one party. When I get time, I'll post my concerns about what's happening to the executive branch agencies that so many people depend on whether they realize it or not. In my opinion, we're facing serious challenges that were largely created by incompetence and inattention, but that have some ideological and self-interested components. I have lots of ideas on how to meet those challenges, too.

By the way, I do have a fairly active online life. This is my first blog, but I have a LiveJournal and have been a prolific contributor to several bulletin boards for more than eight years. The recent EZboards fiasco permanently erased well over a million words (by a very conservative count) that I wrote, not that I have any illusions about the likelihood they'd have been read again. I don't expect anybody else to read this blog either. If I like what it develops into, I'll retitle it and try to promote it.

Why Not the Best?

I may be the last person to have a blog, but I won't be denied. My genius must be shared.