Wednesday, June 26, 2013

My Six-Week Descent Into Madness (aka. working for company X)

Stop, don't do that, it will hurt... these are things that I tell my two year old boys on a regular basis. Sometimes they listen, sometimes they don't. When they don't listen, they often get hurt and hopefully they will eventually learn that when I tell them to stop, it is because I don't want them to get hurt. (caveat: yes, I stop them from causing serious injury to themselves... so let's not get worried)

This is the principle out of which is born recommended practices for businesses in all sectors and in all aspects of a company.
There is GAAP for Finance, HRCI for Human Resources, PMI for Project Management (though Agile practices are probably a better fit in this space), and so on...

These practices do not come from people sitting around thinking about what might happen in situation <abc situation>. They *do* come from actual experience and a lot of collective experience.
Many of these practices are born of principles that come from an understanding of human nature based on observation across many types of people in many types of situations.

Utilizing these practices isn't "going with the flow" or "following the crowd" as some might assert. You are using the collective knowledge of thousands of hours of people doing things the painful way, so that you can stand on their experience and avoid that pain.

I have worked at a number of different companies over the past 18 years as a business professional. And all companies have dysfunction of some sort or another and all have areas of their business that could benefit from adopting proven principles & practices.
Things that would ease their pain and help them be better companies, make more money, and have happier employees.

So now, enter Company X...

Company X is broken. And by broken, I mean, on every level, in every department, in every way. Forget recommended practices... they are following the worst possible practices.

Sounds harsh? Exaggerated? Over-stated? ...I'll let you decide.

I was brought on to this company to help move their software (yep, I'm a Software Engineer) off of its ancient technology platform to a current, supportable platform.
I was told about some of the technologies they still had in active use, so I was aware of the categories of work that needed to be done.
It's the scope that is so massive and daunting, that was so eye-popping. They have over 180 individual applications running to support their business... dozens and dozens of which actually fill the same business need, each one is just customized slightly for a particular client.
Maybe that doesn't sound like a lot... so let me put that in perspective for you. Where I worked several companies ago I was on a team about the size of the team I am on now and we had full-time, could not get to everything work with less than six (6) applications.
There are many many thousands and thousands of lines of code that translates in to hundreds and hundreds of forms/screens/pages of applications that are un-maintainable and un-supportable.

Now, you may be saying, "Ok, that's what you signed-up for, so you just have your work cut out for you." If that were true, I would not be writing this post. I would gladly welcome the full-steam ahead migration of these applications to a common platform. It would be a lot of work that would take years to fully execute, but I'm up for a challenge and a lot of hard work. It's what I do, that's what I like.

But that's not the problem here. Instead of working on these strategic initiatives to get themselves out of the state that they are in, they assign every person available to keeping their broken systems running. Now, I do understand that you need to keep things running while you fix the larger systemic issues. The problem comes in when you keep following the same business formula that got you into the mess you are in.

See, this company grew so fast that they didn't have time to install any safety mechanisms in their business to handle failure scenarios. They moved forward at breakneck speed, hoping to not crash and in a vehicle that has no rollbars, no seatbelts, no airbags, no brakes, no windshield, no helmet and no goggles...
Every time they had a new client, they wrote new applications with new staff/consultants that mostly hadn't worked on the previous ones.

"Get it done, we'll cleanup later." has been the marching order since inception, some 18 years ago. 
Except there has never been any time allotted for said cleanup. They have created a huge bill of technical debt and it is now coming due, but the price is too high and they can't afford it.

Alright, let's step back out of the technical side of things for a little while and talk about culture and process in the rest of the company that I've seen thus far.

Fear
This is one of the biggest issues I see here, it's pretty common in companies to have fear on some level of punitive action being taken which leads to blame assignment. This is definitely at a heightened level here at this company, the fear of "them" and "they" reigns supreme. I can't tell you how many times I have been told some variation of "You might want to be careful about that" and "They might not like that" My response? "Why should I be careful?" "Who is they?" 
This is perhaps due to the fact that this company likes to assign blame and fire people. 
Examples:

  • They have no idea how to manage consulting resources and more often than not, they end up firing one consulting company and hiring another. All with the same people managing them and expecting different results.
  • They fired a whole slew of executives and shook up the org structure just this year. (I'm not saying they shouldn't have been, but the way in which they were, was rather interesting.)
  • They had a business model that outsourced pretty much all their IT functions to a large vendor and then when they realized it wasn't working (1 year into a 10 year contract), they paid big $$ to break the contract and re-hire and new-hire a whole bunch of IT people.
A culture of fear and assigning blame comes straight from the top of the company and has to change there before it can change anywhere.

And change this company must.


Permission-First
There is also a culture here that prevents taking the initiative and trying to come up with new ideas. Again the fear of backlash and failure weighs in on this too. Instead of recognizing that no advances in capability come without failure, the failure-is-not-an-option approach is used and thus no innovating takes place.
Empower your people, people!

Business Efficiency
This is an area that this company excels at not doing. "Let's take business practices from the late-80s and early 90s and use them in 2013... what could happen?"
Paper, paper, paper... the consumption and use of paper at this company is at an all time record. Everything is on paper: notes, minutes, agendas, schedules, plans, emails, word documents, spreadsheets, etc. etc. It's all printed out... single-sided full-size!!!! OMW. And everyone carries around and manages various levels of paper loads. 

  • Taking notes in a meeting? get your spiral-bound notebook and pen out and write it all down... wait slow down, what did you say, I'm trying to write here.
  • Need to reference those notes? just use your handy sticky tabs you put in your notebook to highlight something important.
  • In a meeting and need some information that is only available while sitting at your desk? no problem, schedule another meeting to discuss that information later after you go back to your desk and "investigate" instead of pulling up the information right there in the meeting that you are already in. What happens when you get another question and don't have the information again at the next meeting? rinse & repeat and bring more email printouts, that will help, definitely.
Have they heard of laptops? Not just for managers and certain individuals... for everyone. Everyone has access to all the information at any time and productivity really takes a giant leap forward.

...so much more to talk about in this area...

Information Silos / Process Boundaries
Here we go, this is a good one... there is no company Intranet. LOL! Wow... I actually did laugh out loud about that one when I found out... really, do I need to enumerate the reasons to espouse a centralized, up-to-date location for information about the company to your employees?
There are some sharing locations for information... but you have to know the links to get to them. So, you have to know that something exists before you can discover that it exists. :| And why don't people have enough information to do their jobs?

Oh, oh... so they are really good about hiding information and keeping that to themselves. But really, really bad about setting up process boundaries and following them. Process boundaries are like fences, they make good neighbors. When you know who is responsible for what part of the process, you don't have to keep worrying about that part of the process, everyone can take ownership (if empowered to do so, see comments above) and things get done because people are focused on what they need to be focused.

Conclusion
So, I came here to implement change and everywhere I turn and everyone I encounter is afraid to really change. They keep doing the same things they have always done and hope to get different results. (fyi - that's crazy)
And many (most) of them know it is bad and know it should change, but no one is willing to do it.

This company has had many crisis situations come up that were big red flags to say, "wait, stop, don't do that, fix this" and has yet refused to change. It is my assertion that they are heading for a crash. A hard, up-in-flames, melba toast, crash.

Why do I say this? (see entire post above) And I didn't even mention that they have no redundancy in any of their systems, none, zero... they have infrastructure single points-of-failure ALL over the place...

Oh and they have some very, very large clients that if they left, they'd pretty much have to do drastic layoffs and/or close the doors.
Did I mention that one of their large clients threatened to leave if the company didn't provide Feature X by a certain date? No, oh yeah, that's a good one too... so they slammed everything into panic mode, AGAIN, and are implementing yet another incompatible, highly-customized, one-off system to accomplish what the client wants. :|

Since I came here to implement change, not to convince everyone to actually take a baby-step forward and start the change... the arguments for staying here are quickly becoming untenable.

-sigh- I was really excited about this place and saw a good career advancement/resume building opportunity. But I'm realizing that I was sold a bill of goods and that they were just desperate to get warm bodies in to keep things running.


The stress is tremendous and there's no benefit. Get this, one of the top 3 things I listed as "why I like working here?" - Weekly Friday Lunch... omw! That's the best I could come up with?

Time to check out and punch in somewhere else? Probably.

Tuesday, February 06, 2007

Out of Memory

Have you ever had a problem in Windows XP where you can't open any more programs, but not because you don't have enough available system memory?

I'm talking about the situation where you cannot even right-click on a program in the taskbar and get the system menu to come up. The CPU utilization is fine, there is plenty of RAM available, but nothing works not even clicking on a notepad shortcut. It's dang frustrating!
The only thing that seems to work is to close opened programs. Once you do this, suddenly things start to work again.

You may be saying to yourself, "What the heck is this guy talking about?!". Well, I'll explain. As a software developer, I find I have the need to open many programs at once. Anything from several instances of Visual Studio (2003 & 2005) to browser windows to explorer windows to management windows, etc. And once you get to a certain number of programs, the operating system will stop functioning.

I'd better explain a few concepts before I get too deep into the discussion.
Everytime you open a program, every visual item on the screen is known to the OS (operating system) as a "window". Take for example the very program that you are using to view this blog (if you are using a Microsoft OS) whether that is Firefox, IE, Opera etc. etc. In the active window there are actually about 15 OS "windows" (in IE 6) that are being rendered to create the cohesive interface that you are using.

Here are some image examples:


andandand



These are all examples of what your operating system considers a "window".

Each window is allocated memory, so the more complex your application (Visual Studio or Outlook) the more memory is required for all these individual "windows" that comprise the interface you are using.

So your system has one or two Gigabytes of RAM and you're sitting there thinking, "No way this would happen to me!" Well, you would be mistaken.

See each "window", icon and string are allocated memory from a very specific location. This location is known as the "desktop heap". This special memory location is set to a specific size by the operating system and as soon as you run out of it, bam! you are toast.

(If you want to deliberately experience this specific type of Out-of-Memory condition, you can reproduce it by doing the following: create a shortcut to Notepad.exe or Calc.exe on your quicklaunch bar, click on this shortcut repeatedly until you can't launch another window. You may have to launch as many as 100-200 instances depending on how your system is configured. If you are thinking that I must be crazy to run so many windows and of course it won't work, I'll explain that later.)

In XP there is the concept of a desktop. "So what", you say, "I know that." But what you may not know is that there are different desktop areas, three (3) to be precise. Global, Interactive and Non-interactive. The first is a place where shared system settings and other system memory values are stored. The second is the desktop area with which you are probably most familiar. It is where all visible programs are displayed. The third is where all non-visible programs such as Windows services are run.
Additionally there are by default three "desktops" in the Interactive area, the Winlogon, the Default and the Screensaver. Each one of these desktops are assigned a separate portion of memory based on the configured value in your operating system.

***Caution Note***
If you want to increase the amount of memory allocated it can be done, but must be done with extreme caution as the parameter is extremely important for some basic system things and will prevent your operating system from starting properly.

The key that you are looking for is:
System\CurrentControlSet\Control\Session Manager\SubSystems\Windows

In this value there is a SharedSection=xxxx,yyyy,zzzz portion that you will be particularly interested in. x, y and z correspond to the desktop areas that I described above (1,2,3). By default these will probably be something like SharedSection=1024,3072,512. For most users this is more than sufficient. Microsoft does not recommend setting the second value higher than 5120 (5MB).
I currently have my system set to SharedSection=2048,8192,1024. I have been running with these settings on multiple systems for over two years.
Changing these values will obviously require a restart of your system to take effect.

A couple of caveats, the total memory for all desktops cannot exceed 48 MB. Also the more memory you allocate for the Non-interactive setting, the fewer services you can run. This is also true for the Interactive setting, which limits the number of Terminal Services sessions that you can run.

You may be again asking yourself the question posed above, "How would you ever practically need so many 'windows'?" Well, as you run multiple instances of more complex programs you start to use a good number of windows. How many you ask? Well for instance Visual Studio 2005 with a small solution open and two files consumes 186 windows. And each additional opened file in VS 2005 consumes more than 70 "windows". IE 6 with this site loaded consumes 16. So as you work on multiple projects of greater complexity and have multiple IE windows open and Notepad windows and Windows Explorer windows, you start to bump into the ceiling.

My experience is that 512 KB allows for about 1500-2000 windows depending on other system conditions. With the default setting of 3192 this allows for 9000-12000 windows. Again for most users, this is sufficient, but for a developer it might not be. With my setting of 8192 this allows me to have a window load of 24000-32000.

My experience is that 512 KB allows for about 1500-2000 windows depending on other system conditions. With the default setting of 3192 this allows for 9000-12000 windows. Again for most users, this is sufficient, but for a developer it might not be. With my setting of 8192 this allows me to have a window load of 24000-32000 which is plenty for what I need to do.

If you want more details, please see the links below, where I obtained most of my information in addition to the information that was obtained from Microsoft Technical Support.

http://support.microsoft.com/kb/126962/
http://support.microsoft.com/kb/184802/

Monday, February 05, 2007

Blog Kickoff

So I have had this blog for a while and have done nothing with it. I'm going to change that... so expect some new posts here very shortly.