The Magical World of Software Development (Part 1)

The software development world is wild. I would argue that no other industry is quite like it. Its half art half science. Some people go to school for it, some don’t. Some people get it and are on a completely different level, and some people struggle through and just make a living off of it. I have a theory on why all of this is. To explain it, I have to go through the history of software development as I see it.

The Early Days

Computer Science used to be just that, a science. Way back in the day you had to understand the physical makeup of the computer to be able to program it. Scientists were working with resistors and gates and all kinds of things that I only have a basic understanding of, because I am not a computer scientist.

But it was a practical science, not a theoretical science to the people who participated in it were called engineers. They used engineering principles since they were dealing with the physical world.

Think of how physical structures are made. An architect designs the whole thing. Engineers are involved in modeling and stress testing the models according to their specialty and then when it moves into the build phase manual workers build the structure according to the specification.

Because the software was so dependent on the hardware, software engineering was a physical science.

Software as Engineering

As computers advanced, engineers were able to build more complex applications. The coupling between the physical computer and the software running on it was loosening.

Even as there became less need to understand the physical structure of the computer, software engineers still applied processes from the physical engineering world to act of building software. This is evident from the major development lifecycle created for software, waterfall.

Waterfall mimics the processes of creating physical things. Architects designed it and engineers planned it and built it. Software development was more like manual labor. By the time the code was written they were just implementing designs and specs that we already determined, committed to and meticulously specified.

People kept building software like they were building bridges. There was long planning processes to make sure everything was thought of and all aspects were considered. The specifications were detailed and left nothing to interpretation. Only then would work begin on the project.

But something happened. As languages evolved and became higher level the coupling between the hardware and software that had been loosened finally broke.

The problem was that engineers were building in a digital landscape not a physical one. And by the time the thing was built the landscape had changed. Imagine building a bridge over a river and by the time you are done the river was twice the size and a mile away!

We needed to leave the rules of physical engineering behind because we were in a new world. A world that needed a new set of principals. A world that needed a new kind of worker.

We needed to build things but also adapt to the way things were changing. Enter Agile.

The Magical Digital World

The digital world is like a magical kingdom. And once people started to understand this the rules and the tools changed.

Lets go back to the bridge example. Imagine if you wanted to test something about a bridge so you built two of them, or a hundred of them. Maybe you wanted to test if people enjoyed one bridge more than another.

None of that is feasible in physical world. There is a small bridge that goes over a creek near my brothers house. Its currently being replaced. It is a simple bridge but it will take 6 months to replace.

The digital world doesnt have the same rules as the physical one. We can simulate everything on the real infrastructure of it or exact replicas. For example we can simulate traffic, infrastructure outages (chaos). We can swap out the infrastructure with new infrastructure daily or even multiple times a day.

Imagine doing that with a bridge.

At some point we realized that the physical rules didnt apply. We were building using engineering processes designed for the physical world but we were in the magical digital world. It was inevitable that our processes would change.

Agile

If you follow the history, the realization that we were “not in Kansas anymore” wasnt like flipping a switch. People began to realize it slowly. There were frameworks like Kent Becks Extreme Programming or James Martins Rapid application development. I am sure there were lots of others. These were people who didnt accept “the way we always did it” and began to see a new path forward in the digital world.

If there was a switch to point to it would be the Snowbird Utah meeting where a lot of these people got together and came up with the Agile Manifesto. I wont go into the almost Arthurian tale that has spun up around that meeting. It was important, the people were smart and yes it changed the was we think about software.

Agile took the architect, engineering and code writing process and shortened them to small iterations. In order to build in this new landscape we needed to be able to change the design quickly. This took the 6-12 month timelines of software projects and broke them up into 2 week increments. This meant that the phases couldnt be separated and the people didnt need to specialize on a certain phase. The same people could architect the solution, test it and build it.

The software was built incrementally so that it could adapt to feedback and to the changing landscape. And the landscape was changing fast. Projects could no longer take 2 years because the world they were released into would look nothing like the one that existed when the project started. This feedback loop meant that the world began to change even faster.

Naturally people’s tolerance for error went up. Without laying out the design completely ahead of time keeping everything synced up was difficult. But developers began to realize in most cases the consequences of errors in the system were not as they were in the physical world. If you built a bridge that didnt connect in the digital world cars didnt go careening off it.

The Present Age

That brings us up to present time. People have fully embraced this new digital world of software development. With the advent of the cloud it took things to a whole new level. Software was completely untethered from the physical world and now not only could the software be created in this magical realm but so could the physical components needed to run it. Working software could be delivered in weeks or days.

This has lead to more pressure on the timelines. Everyone is moving fast and the only way to win is to move even faster. With the new AI models entering the development world and the general population it is getting even crazier. Writing software isnt just untethered from the physical world its now disconnected from the logical one as well. You cant look at the model and see the flow of logic through the software. Its truly magic.

Part 2

This post was originally about how to manage software developers. As I worked on the title I was trying to decide if I should call it “How to Manage Software Engineers” or “How to Manage Software Developers”. I went to look up the difference and the results were mostly about the definitions of the word “developer” and “engineer” and the conclusions varied wildly.

I also didn’t see anyone talking about how the history of the trade had changed drastically from the time when engineering principles and understanding resistors and gates were required to the time of feeding data you pulled off the internet into an LLM.

Some describe writing code as a science, some describe it as an art.

I wanted to understand who the people were who were writing the code and how to manage them. In order to do that I felt it was important to set the stage for how this eclectic group was assembled. We have PhDs down to self taught enthusiasts in all levels. Just like the rules broke in the digital world, they also broke for who could succeed in the physical world.

In part 2 I will try to break down who the people are in this magical world of software development and how to manage them.