As the facilities director for schools on campus, among my responsibilities was the infrastructure for the buildings in my domain. Here is the story of the Great Generator Caper aka much ado about nothing.
One of the original buildings on the campus, built in 1969 or so, was designed (Periera) and built as the first University of California Computer Sciences Laboratory building. At the time, computers were gigantic and took up entire rooms if they were any good. The power needs for areas that supported “Big Iron/Big Blue” systems was pretty immense and it was quite expensive. UC was taking a huge risk betting that one day, computers would play an important part in education, not just commerce...their most likely usefulness.
Well designed buildings assume future growth and in this case the plan was for more big power-hungry computer systems. The resulting design was aimed at handling the Power and HVAC needs of adding at least one more “Big Iron” computer system just like the first one.
So here's the thing… over time, advances in technology occurred in a much greater speed than expected…exponentially so. As years and decades went by, computing changed from one or two ”Big Iron” Machines in a single centralized location in a computer lab building to today’s hardware and software platforms located *not only* on every desktop but ‘in the cloud’ too.
Physical CPUs and computers have migrated from ‘Big Iron’ to become racks of servers…fancy high-end computers running Sun, Vax, and others in a centralized location and accessed via “dumb terminals” (e.g. the vt-100) on some desktops. Then the computers/CPU’s started moving to desktops. Now the computers all sit on remote desktops and/or The Cloud and the old Big Iron-cum server rooms have become racks of massive data storage devices that require a much heavier HVAC and heavier power load then even the Big Iron machines had needed.
With their ever-zippier CPU processing speeds and ability to pass data at rates as high as 1GB or more, the infrastructure of a building only 5 years old could quickly become obsolete for corporate R&D work…everything we had in place was getting to be 40+ years old.
The massive amounts of data are passing through switches, hydras, UPS’s and ethernet wire (Thin-net, 10BaseT and Cat 5 at *best*) were designed to handle much fewer datum to and from centralized data storage locations as well as in ‘the cloud’.
Power requirements and power distribution have changed significantly over the decades. The Heating, Ventilating and Air Conditioning (HVAC) requirements have changed significantly, from supporting large rooms with fewer people to small rooms with a lot of people and machines generating heat and consuming power. Sizes and shapes of what ideal efficient computing “laboratories” and classrooms have changed significantly over time, as informatics, ergonomics, data management, and other systems have been developed.
Effectively, the oldest buildings’ capabilities and ability to function using the progressive forward-thinking design had become obsolete. As ‘laboratories’ changed and rooms with seating and chalkboards/whiteboards where theory had been taught and keycards punched to big rooms with rows of chairs and tables that had computers locked in place on/in each one. Don't even get me started on the evolution of computer hardware security, it’s also evolved.
The original buildings on campus were all built with an emergency generator in the basement or other ground-level room designed for this purpose. In this case, the generator was one of those things you see in a room in movies…a big-ass 12 cylinder diesel engine that was about 10 feet long. It could handle *way* more than the projected needs during the design/construction of the building. They all did back then.
Over the years it became apparent to the individual people in charge of getting things done in specific buildings that the University was all lovey-dovey up front during design/construction of university properties, then washed their hands of the whole thing.
Until some point in the ’70’s (I believe, for sure by the late 80’s), funding was apportioned to each campus annually for planned maintenance projects to pay for planned maintenance and upgrades that would be required by any specific building over decades.
That started to dwindle into a special “deferred maintenance funding” program, leaving infrastructure maintenance to good fortune and chance. If the roof doesn’t leak before 15 extra years after it should have been replaced, it might come up on the deferred maintenance projects log, distributed by the good folks in the IOB building one time per year in July. If it did…great. If not, we’d cross our fingers and hope the deferred maintenance program didn’t expire before the building got a new roof or elevators or whatever.
It’s clear as day to the people who’ve been plugging the dam holes over the last couple of decades that financial support has decreased to levels requiring a different kind of planning in advance.
When the deferred maintenance program did expire, it was replaced with a culture of ‘plugging holes in dams’ until you could get a tradesperson to actually show up to fix anything. It costs *much* more to it this way and once that slide begins, the culture of ‘costly-emergency-maintenance’ slope started getting real slippery.
Think of it as health care for the buildings. Right now for people 60 years and older, doctors know and help plan for upcoming and/or future ailments, diseases and so forth. Knees wear out, hips wear out and/or break, osteoporosis, sun exposure, nutritional needs…we know about and plan for this regular stuff as it happens to us. Wealthy people can address these in advance or contemporaneously as the issue progress. Poor people have to wait until they can afford to pay the doctor and/or the issue to get acute enough to need a trip to the emergency room.
Buildings are like people in this way. It’s easy to plan critical/crucial electric/plumbing/roofing/etc. upgrades that are needed years in advance. You cross your fingers and keep saving up money until you can afford it…but your Dad keeps spending the money on hookers and blow. And he’s Dad, so…
Still, if you cannot afford to call in the contractors, the building systems get sicker and die.
Ok, back to the generator. I’d been doing upgrades to the building as we could afford them oh-so slowly, when a grant proposal I wrote to upgrade the power in this 1969 building was funded. That funding program is a whole different kind of crazy that I can cover in another story.
It was my laser, used to surgically repair the worst specific illnesses any buildings suffered from. A weird laser with control knobs that change by themselves, sometimes during use. Passwords to unlock the funding needed could change mid-stream on the whims of unseen controlling forces. Very little was certain to remain static, and the changes were often drastic or draconian.
There was this guy, I’ll call him Guillermo, who came around to each building once a month to test the emergency generators. He’d come in around 5am, fire up the generator and let it run for 20 minutes or so. There was a hand-written log book at each generator location, Guillermo always wrote the details of the exercise he performed. He wrote the date, time started, time run, condition of belts and consumable parts, the stuff a person unfamiliar with that specific piece of equipment really would need to know in order to quickly/efficiently diagnose and repair problems. It’s required by law and it’s always filled in. Guillermo is fastidious in running and recording the monthly testing of the generators.
As the person most directly responsible for the physical *buildings* (and the contents) that the generators are placed in, it was prudent for me to know the maintenance/repair status of the building systems, including the emergency systems. In a conversation with Guillermo one day as the giant 12 cylinder generator was running in the background, he mentioned that he’s been doing the testing for 40 years. I cleverly calculated and induced he’d been doing the testing since there were three buildings on campus. He knows generators. He knows *these* specific generators upside down and backward. “You know what the problem with this generator is?” I replied “No, tell me”, expecting to learn of a fatal mechanical flaw real or imagined. “This generator doesn’t have enough power on it”. He went on to say the generators are ‘rated’ to run x-number of hours, that the electronics…capacitors and other parts…need to be “exercised” more. Was he telling me they needed a bigger test-power load put on them regularly b/c they might not handle the entire building/zone needs when actually under use for extended time? Yep.
This was interesting information, and it made sense from my high school electronics level perspective. I quickly wrote a grant proposal to upgrade the total power voltage to the building and upgrade the distribution power network of the building’s systems to handle the newest methods of distributing power, data and HVAC. We planned to use the building for this same purpose for many years to come, so how about it? The grant was funded and I had about $250,000 to do what I needed to do.
I added big-ass power feeds to building, bucky bumper transformers at centralized locations on the affected floors leading to new 250 amp circuit breaker panels. From there, I could move power around any specific floor(s) of the building as needed at a relatively small cost. As justification, I wrote a memo about what Guillermo had told me about the generator being underworked. Justifiable improvements were golden tickets in the ‘what’s going to get fixed’ lottery type system. This project was funded, completed on time and under budget as usual. The generator was under a much higher load during testing and everybody was happy.
After work about 3 months later, I was driving to the university’s sailing base when my phone rang. “Hello?” I asked as I enabled the bluetooth handsfree speaker. It was Anne, my boss and Assistant Dean of the school. “Just touching base with you about the generator problem…”. When I asked what generator problem she was speaking of, she seemed surprised.
It turns out Anne was in a meeting with her peers from other schools on campus and one or more Associate Vice Chancellors (AVC’s). Sally, a special assistant to the AVC for Facilities Management (for the entire campus macro operations). Sally had mentioned that I had told her that “our old generator was tired and worn out, needs to be replaced”. Sally was lying in an attempt to extort about $200,000 from my school under the guise of “splitting the cost” of $400,000…and she was being all lovey-dovey about it. When Sally was happy, somebody else was getting screwed.
So there I was, driving in my car having to proxy-battle with batshit crazy lying Sally’s claims about a problem she invented for God know what reason. This was Sally’s M.O. for about 5 years until she was ousted during a shifting administration when her mentor/supervisor/personal friend (the AVC) moved up the ladder to a different division on campus.*
I’ve just explained the actual condition of the generator system with pretty good details about the situation. This was nuts. I asked Anne to let Sally know we’d get back to her on Monday…today was Friday and I was on my way to race sailboats. Then I probably won the race and never thought about any of it for the weekend. Heh.
On Monday, we set up a meeting with the AVC to talk about this situation later in the week. When we showed up at her offices, we were directed into a room where Sally was sitting. She was alone and said the AVC, Mary, was tied up and we could work it out with her instead. We got up and walked out of the room, drove back to the campus being amazed the we were amazed this was happening (again). Or still happening. Or something. Nobody was stopping the reign of terror.
Anne and I had a working relationship with Mary, developed separately over 20 years or so. Mary was a straight shooter. At least she *was* until Sally came into the picture. She still was mostly, but it felt like she was always giving cover for Sally’s shenanigans.
Sally had apparently told Mary that this was just a small details meeting, not to waste her time on it. Neither Anne nor I were surprised. Nor would it have been a surprise to most of our peers on campus…this was how Sally operated.
Sally had an even worse flaw than her lying…she inflated costs/pricing on everything she handled by at least 100%, and she was suddenly promoted to supervisor over the other project managers so there was nobody to complain to. Her boss, her personal friend and confidant, had to *believe* everyone on campus when we reported this…we’d been working together years before she brought Sally on board…but she never did anything. Impunity isn’t all that rare on campus. Probably felt awkward for Mary, sure felt crappy to the actual schools on campus, who needed improved facilities to teach and do research.
Regarding the way a university campus operates, the easiest way I’ve come up to describe how things work is to say the university is a state, and each of the schools are cities who operate independently but in cooperation with the university…who has a death grip on funding for almost every school on campus, some schools are independently wealthy and some will never be wealthy, solely because of their social status in the academic community.
There are also ‘casts’ of people on a university campus: There are:
*Well-published faculty
*Faculty who teach giant classes with thousands of happy students behind them
*Faculty that haven’t been publishing enough articles with the campus’ name on them (death spiral).
*Staff who are untouchable due to their power and control over the entire university
*Staff who are untouchable in their own school (e.g. Physical Sciences, Humanities, Social Ecology) due to personal relationships and/or ability to get things done who are vulnerable at the university level
*Staff who aren’t very vulnerable because they get shit done and their boss has their back.
I sort of fit into the latter category, I got things done and had working relationships with people in most other academic and administrative units. I had respect because when I called BS it was BS, and it was generally in a hammer-blow fashion. No word mincing when people are lying or making stuff up. “That is not true.” “These aren’t real numbers, here are actual numbers I just got off the internet.” “Don’t tell me you have never been responsible to fill tampon machines in the restrooms when all that really happened was you stopped because you weren’t making money on the deal.” If I were wrong about this stuff, I’d have been very vulnerable at the university level. If the Assistant Dean of my school didn’t have my back, I was vulnerable at the university level. I was careful to work for Assistant Deans who had my back when I was switching from one school to another every ten years or so.
Back to the generator. The generator that had been fine for 40+ years that the guy working on it for 40 years says is healthier now than ever before. I started making inquiries of my peers in other cities to ask if they’d heard anything about generator shenanigans. I had been Chair of the group for 8 or 10 years along the way and had working relationships with all of them. They knew nothing. A month or so later Sally set a meeting to discuss it with the AVC’s Project Management Team and I was invited. I was going to attend.
In general, the idiocy/ignorance of groups/committees on campus who plan and implement expensive changes and/or upgrades in tricky locations without thinking it can be maddening. It would be a good idea to have a hands-on person who actually knows the ground status, has a stake and will literally be responsible for the results of the plan into the room during the planning stages is almost untenable sometimes.
The meeting.
Anne asked if she should be at that meeting and I told her “I got this…that’s why you pay me the big bucks” and we had a chuckle. I drove to their off-campus offices called the “Interim Office Building” that we called “IOB” because it was put there in 1968. That still qualifies as interim on a university campus timescale. I entered the building (mobile home) through the back door and said hello to the staff who handle trouble service calls for the campus, my lifeline into ‘what the hell is really, actually happening’ because the tradespeople are being told to lie to us. I delivered the Trouble Desk tins of expensive cookies for the holidays every year.
I turned the corner to enter the project management cubicle farm and see a conference table wedged into the center of the room. Sitting were 2 of the university project managers, 2 outside electrical contractors and a couple of other university mucky-mucks sitting around it.
The Project.
I joined the group, who were just starting to lay out schematic drawings/blueprints and proposal papers and I waited for the ask. Why was I here?
“Our qualifications are…blah blah…”The campus has hired us 3 times before”…blah blah. This started to feel like a bigger job than replacing a perfectly good generator in one of “my buildings”. I started looking over the blueprints and drawings and didn’t see what I expected to see. I asked what we were doing, exactly what the scope of this project was, to set up parameters so that I could bring the meeting back to topic when there were too many words being used incorrectly. I moved in for the kill. “What is the bottom line cost on this project?”, I asked. One of the mucky-mucks tried to jump in to muddy up the waters, but was too late…the contractor had begun answering…”$3-5 million bucks based on decisions made at this meeting today”.
OK, this was bullshit. A dog-and-pony show to extort schools for money to pay for work the university is responsible for. They’re paid to be responsible. They get money from the schools and from Campus Administration to cover this stuff. For instance, 60% of any faculty grant is skimmed of the top to cover “FNA”, which I don’t think actually stands for anything…it’s what normal places call G&A, General and Administrative costs. Power, water, cooling, heating, decently clean walls and carpet, desks and chairs, staff hours…stuff like that.
I took a mental stand. If they’re upgrading a bunch of generators to score some ‘LEED Points’ with California or the Federal Government, they are welcome to upgrade anything they want…on their dime. My generator works fine, thank you very much. I started to look for an exit strategy. I wanted nothing to do with this, I knew what they were doing. They were trying to scam me into supporting the project, so they could tell the rest of the schools that I did. I had a no-bullshit reputation all over campus and they wanted my buy in without expressly asking for it. Had they offered to do all the work in my buildings at no cost, it would have been a bitter pill for me to choke down…but they didn’t. They tried to bamboozle me with bullshit instead.
I don’t have no fancy college degree or nothin’, but I know when something stinks and I know it stinks from the head down. If a project starts with fancy suits and questionable premises…BS and lies, it’s not going to get any better along the way…running like hell is a good tack to take.
I was done there, I could be doing actual reality-based things back on campus. It was neutron bomb time. I opened the folder I had brought with me and pulled out the 3 copies I’d made, so they didn’t have to share. Turned out to be more people than I expected at the meeting, so they had to share. There’s only so much a guy can *plan* for. It was a single sheet of printer paper with the catalogue price of a Caterpillar Generator with a higher capacity, mounted on skids, so it could be deployed from the back of a truck and slid right into place. The list price was about $63,000, but I was polite and told them I hadn’t called to haggle yet. I told them I’d gladly pay to replace the generator with *this* one or any other one available “off the shelf”. I was done here, asked if I was needed for the rest of the meeting before I dropped the mic and walked out the door.
A couple of weeks later, I bumped into a university project manager one day at lunch…the honest one of the half dozen or so on campus…the one who was not included in the afore-mentioned meeting. Out of the blue, she mentioned a new California regulation requiring a specific type of ATS (Automatic Transfer Switch) and wiring (etc.) be installed on emergency generators, even retroactively. She told me the campus was putting together a project to replace all of the switches and control panels on all of the generators on campus. We chatted a while and I left, with critical information that had been hidden from me and my school.
It’s not cheap to replace wiring/switches/panels/whatever else that were legal in the 1960’s with new university-of-the-art power transfer equipment. My guess was that it was going to cost the campus about $150,000…the same amount Sally had tried to scam from us to “fix” our beautiful 50 year old cast-iron generator.
Ta-Da! There it was…a fact that made sense and explained the project I saw on the conference table. I reached out to Sally to let her know I’d offered to pay for a replacement generator which comes with the new ATS and wiring section. If they refused to upgrade the ATS stuff for the ’60’s until CA passed a *law* to make them comply, my budget was off-limits.
They replaced the electronics at their expense and that is the end of the story.
*Fun Fact: I recently learned that Sally has recently returned to campus, 15 years later, in an even more critical role, in the unit that plans and puts funding in place for new buildings starting about 7 years before construction through about three years of construction and outfitting the finished product…in 10 years. My peers who are still on the job are horrified and disgusted, at least one planning retirement rather than fight Sally’s bullshit for any longer than absolutely necessary. Sally lies. Sally lies *a lot*. Big lies, small lies…didn’t matter. In retrospect, Sally’s kind of Kavanaugh-like (dead serious) in many ways…including landing a job wholly at the opposite end of the truth/fact spectrum from her working style.