I wrote 100 million lines of code and all I got was this lousy t-shirt!

People who know me probably know that I’m not a huge fan of too many people – certainly not Elon and not Sandy either. I tend to be a bit critical of public personalities, especially those two. But, the above video is a very good discussion about the state of the automotive industry and how badly most automakers have bungled the software so far. Somewhat interestingly, I don’t really include Tesla in that. They’ve been doing a much better job than anyone else. Kudos, Elon!

In the video it is mentioned that cars these days can have 100 million lines of code running within them. However, I think that measurement is a bit meaningless. They use MATLAB / SIMULINK to generate code. Because large stretches of the code are auto-generated it doesn’t really matter how many lines there are, nobody wrote them. Also, that would include all of Linux when they use a Linux based display or computer. And, if you have two Linux machines in the same car you add the source lines for the Linux kernel twice now. Likewise, there is a LOT of boilerplate code for bring up of the processor and other hardware. Each ECU probably has its own RTOS (real time operating system) which adds lines but it’s all pretty much the same RTOS over and over for each ECU. So, people like to trot out 100M lines but that obfuscates how much of it is actually used to develop a car and how much of it is support infrastructure. As an extreme example, if you write a program for an embedded processor that prints “Hello, World!” to a serial link that program will be a lot more than “printf(“Hello, World!\n”);” the processor clock must be initialized, the RAM has to be configured, the serial link hardware has to be configured, etc. It will be thousands of lines of code just to print a single line. The Linux kernel is several million lines of code all by itself. So, I don’t think saying “100M lines!” really does anything. Yes, there are a lot of lines of code running within a car. And, yes, more code means more places where someone could have made a mistake. But, largely that code is comprised of duplicated code, boilerplate, and things auto-generated by a computer.

They spent a lot of time talking bad about C and how it is old. We don’t use 40-60 year old languages that were frozen in time like some caveman. C has gone through multiple revisions to the standard and most automotive code uses C++ which is newer still. C++ didn’t come about until 1985… OK, bad example, that’s nearly 40 years ago… Common C++ is up to at least C++14. Newer versions exist but usually C++11 or C++14 are targeted still. In this case, the 11 or 14 means 2011 or 2014. C++ continues to evolve with the times. Writing C++ code today is not the same as writing C++ code 30 years ago. There’s nothing wrong with using C++ on a car. It’s fast, efficient, and can talk directly to the hardware. Also, anyone who used C++ compilers in the mid 90’s knows they sucked so much that you had to write assembly routines yourself to speed things up. Today compilers are fantastic and if you write your own assembly about 99.9% of the time you will do worse than the compiler. Technology has progressed by leaps and bounds in 30 years. Real time operating systems have come a long way too. The world has not stood still and I think it’s disingenuous to suggest that the use of C/C++ in a car is somehow antiquated or inappropriate. C++ is what you make of it. You can write modern things in C++. You can also write horrible spaghetti code. Simulink is not likely to write code that is as nice and tailored to the desired hardware as a human would do. But, it can do it in C++ and so can a human being.

However, I do agree that the use of such languages as Rust and Go would be a good idea. Of those two, I belong to the cult of the Rustaceans. Rust is, roughly speaking, a lot like C++ and can be used in many of the same places but does away with many of the ways C++ lets you shoot yourself in the foot. In Rust it will not let you shoot yourself in the foot. The language just won’t allow it. But, it’s a harsh taskmaster and it requires strict obedience to the rules or it will refuse to compile your code. This leads to a lot of frustration at first when C++ developers, used to the Wild West where anything goes, are aghast when the Rust compiler forces them to write code which is more secure. Is that to say that you cannot write bad code in Rust? No. Is that to say that Rust code is always 100% secure and bug free? Heavens no! But, it enforces restrictions which overall are “for your own good.” And, best of all, Rust runs basically just as fast as C++ code so you don’t have to sacrifice any speed, just sanity. What a bargain! All joking aside, Thomas and Sandy are right that the future does not belong to C++ forever. It is indeed old and carries with it the burdens of being originally designed a long time ago for hardware which is long forgotten. Rust is very likely the future for applications which require both speed and security. Wouldn’t you know, the automotive sector pretty much perfectly falls into the sweet spot where Rust is a good idea.

Thomas really seems to like the idea of micro-services. It isn’t a bad idea on the face but they should be running on the car not in the cloud. Too much mention of the cloud makes my skin itch. Micro-services could be written in python, go, rust, or C++. There could exist standards for how they interoperate such that pieces could be mixed and matched. It’s a pretty good idea. Though, perhaps we should take a step back. What is a micro-service and why do some people want them? Traditionally software is written as a big blob of “stuff.” It’s all in there. When you write a control unit for, say, the HVAC you have a lot of routines in there that handle various things – reading the dials to see what temperature you want, whether you are asking for heat or A/C, reading the in-cabin temperature, controlling valves and other systems to try to match the temperature you asked for, etc. It’s all a bunch of routines all baked into one big batch of code. Let’s say I think it’s dumb the way it handles automatically switching from heat to A/C as the temperature outside goes up. I hate it. What can I do to fix that? The routines to control this are buried in a giant batch of code hidden inside a control unit deep inside the bowels of the car. If I don’t like it I’m free to pound sand. And, what if the car company agrees? Now they have a software developer look at the code to see if they can fix it. The code gets called from various places. Once they find the code that controls these things they try to modify it but then they have to test everything and recompile the whole program. What if there were a different way? What if the HVAC system was a bunch of little processes that each handle one simple, easy to understand thing? What if those pieces all talked to each other in some standard way? Maybe one little process does nothing but read the state of the temperature knob and report back the set point. Maybe another does nothing but read the outside air temperature. Maybe another does nothing but open and close a valve in the heating loop. These little tasks can be called micro-services. They’re small, they’re simple, and they do just one thing. Each little micro-service is easy to understand. They have limited inputs and limited outputs. And, even better, if I don’t like the way one of them works, I could just rework that one micro-service and the rest of the system need be none the wiser. Everything communicates through standard interfaces and does limited things so changing any one thing doesn’t really affect the rest. For sure, sometimes a change in one component causes changes in the way connected components operate. But, testing can be limited just to the parts that interact. And, pieces can be mixed and matched. The end goal of micro-services is for each thing to do “one thing well.” This concept could free automotive systems from being so heavily tied together into big balls of goo that no one can change or maintain. Traditionally micro-services have been used in an internet context but I see no reason they couldn’t interact locally within an automotive system such as HVAC.

Thomas talks about moving from Linux to Android. It seems like he doesn’t realize that Android is just a layer on top of Linux. All the Android phones out there are actually Linux machines with an Android GUI on top. The transition needn’t be very difficult at all. But, I do like the idea of consolidating ECUs and building fewer, more powerful devices that have more horsepower. I think that’s a reasonable approach. It’s the approach people tend to take when doing DiY conversions. When you build an electric car from parts you tend to want to control it all from one master controller and limit the # of ECUs to save cost and complexity. Our own GEVCU7 is a very powerful control unit meant to be the glue that holds everything else together. But, ultimately it is still the root of a hierarchy of ECUs which are spread out in the vehicle. The OEMs should probably consider going toward fewer computers which control a greater range of hardware but do so with easier to modify pieces that all run as loosely connected pieces that exist on the same computer.

I agree with them that companies are doing a lot of short sighted things and not making good engineering choices because of pride or corporate inertia. And, yes, quite a lot of automotive seems to be held back by how they DON’T write their own code. Instead they have engineers design diagrams in Simulink and have plugins written elsewhere spit out code that few people understand or read. Making changes is difficult because they didn’t write the pieces they’re putting together and too much is found spread across tons of ECUs. Micro-services won’t necessarily fix that but breaking things into small interconnected systems with standardized interconnects could simplify some things.

Creating standard architectures where the hardware of the vehicle could stay the same while software changes is a great idea. I hope automakers do get to that point. It is possible to do. You can customize Android, for instance. And, Android has standard APIs for hardware access so you don’t necessarily need to know which actual hardware you have when you download an Android app that uses it. It would be excellent for end users if this kind of thing could come to cars. But, this point is somewhat at odds with the above point. They already don’t write their own code and instead use building blocks they don’t fully understand. So, and this will probably be the most controversial thing I say, I think the entire industry needs to form a coalition and create a set of common standards that make sense and then open source the entire thing. Linux succeeded because everyone could work on it and target it as a standard interface. Long has it been argued that the year of Linux on the desktop is “this year.” But that year never comes… or did it already come years ago? Maybe we all looked at it wrong. Android powers more phones than any other mobile OS. And, it’s Linux. Linux underpins many embedded devices, cars, etc. We didn’t get Linux on the desktop, we got it EVERYWHERE ELSE. It won long ago. Open source has taken over the world because nobody can clutch it to their chest. Linux is everyone’s and no one’s. The automotive field as a whole needs to learn from that and make a system where the core is everyone’s and no one’s at the same time. A standard set of APIs and interfaces that everyone uses, nobody has to solely be responsible for, and that everyone can build on. Just like Linux. Even with Android there isn’t just Android. Samsung makes their own skin over the base Android. Others do too. In cars the base can be common but self driving features can still be the “secret sauce.” The way the dash looks, the way the other displays look can be customized by each car company for their own vehicles. In the end this will save huge numbers of hours in reinventing the wheel at each company while freeing them up to do what people really want – to build new cool stuff.

I absolutely agree with both of them about things like protecting kids in a car. There is no good excuse for why this can’t happen soon. I’ve beat up GM a lot over how they refuse to implement good ideas in their cars. For instance, pet mode (which is identical to the child protection they talked about) or sentry mode like Tesla has. Why can these things not be added even by third parties? Only because GM has done a crap job at engineering. But, it doesn’t have to be like that forever. The industry as a whole could push to do better. Both Thomas and Sandy are likely correct that bean counters have ruled the automotive sector for too long and have stagnated companies nearly to the point of extinction.

I have a 1985 Mercedes 500SEC. This car was $56,000 in 1985. The CPI inflation calculator says that $56k in 1985 would be $157,541.54 in today’s money. Now that’s an expensive car! In the 80’s Mercedes just let their engineers play. Essentially they were tasked with building a luxury car. And, build a luxury car they did! In 1985 it had automatic AC and heat, incline sensitive cruise control, heated seats, air bags, anti-lock brakes, fuel injection, power seats, and probably many other innovations I overlook because they’re common in 2023. But, it was far ahead of its time. You paid for all that but they didn’t spare any expense. They built a car they knew was innovative. Around the year 2000 Mercedes had a come to Jesus moment and started trying to make cars more like everyone else. My parents have a 2000 Mercedes. It has some cool things – a navigation screen, air ride suspension. It’s still ahead of its time but the thing is also a piece of junk that falls apart. The air ride suspension will kill you. As a point of comparison – my 1985 Mercedes is still on the road. I’ve had to change some things – replace valve cover gaskets, flush the transmission, replace brakes. But, the car is still going strong. The 2000 Mercedes has been sitting for years because it’s too dangerous to drive and even the Mercedes dealership cannot figure out why. The biggest change within Mercedes was to try to let the bean counters tell them how to build cars. But, bean counters don’t know how to engineer cars, engineers do. You might argue that $157k is too expensive for a car. I’d have to agree which is why I didn’t get that car until the price had significantly dropped. Surely there is a middle road where engineers are allowed to be engineers but someone still reins them in as engineers won’t stop until perfection is at hand. Which, of course, never happens. At some point you just have to ship it anyway.

But, what is the biggest difference between how Tesla operates and how GM operates? Well, Tesla lets the engineers engineer the car. GM lets people with MBAs have the final say. That’s not to say that Tesla does everything correct. All too often Elon’s whims and temper tantrums turn into engineering challenges. But, still largely Tesla seems to understand that you have to let engineers do what they do best. Perhaps engineers need some adult supervision to tell them that building $300k cars isn’t going to work. But, Tesla largely does a good job of delivering the technology people want. GM knows about building cars but they’ve let their bean counters run wild.

Though, absent in the above discussion is the large elephant in the room. Is there a reason automotive vendors move so glacially slow and are so adverse to making any changes? A reason I have conveniently ignored up until now? Yes, and it’s a MISRAble reason. You see, MISRA and ISO 26262 are sets of standards for how to design safe, secure systems. ISO 26262 in particular is a safety standard for designing software in vehicles. These standards are well meaning and likely necessary to give everyone a concrete set of standards to work against. Nearly nobody has a good time in following these standards. Like Rust, sometimes there are some annoyances and sometimes the medicine you need to swallow is a bitter pill. These standards are no exception – they’re a pain to deal with. But, they do allow automakers and part manufacturers to write code which is more secure and safe than it might otherwise be. However, like all standards, the devil is in the details. There is essentially no standard that cannot be used to pave the road to Hell.

There are a few ways standards tend to go wrong:

1. They get immense. Standards are great when they first come out. But, as more and more people need to put their own touch on the standard, it gets more and more bloated. After a while, committee driven standards become unholy abominations before the Lord.

2. They get locked behind paywalls. The big companies don’t bat an eye. But, this locks smaller developers out of the game. Not so coincidentally, locking out competitors sounds amazing to the companies already in the game. Also, standards bodies love to play the “pieces” game. I say ISO 26262 and what do you think? Maybe you can order a standard called ISO 26262? Of course not! They’ve broken it up into six million chunks. You will not know which ones you need and they’re all priced separately. Once again, no big deal for a big name OEM, quite a hassle for the common man.

3. An “ecosystem” tends to build up around these things. MISRA and ISO 26262 are no exception. Wouldn’t you know, there are companies willing to sell you tools that validate your code against these standards? Now, I’m all for capitalism, but making standards pretty much a requirement then blowing the standard up into a thousand pieces, charging piecemeal for the whole mess, and then having $20,000 tools (no, they aren’t all that price, they can be much cheaper) to do the checks is kind of a bad look. If safety is the priority they could create open standards.

I’m hopeful that car manufacturers have learned the lessons they needed to learn from Tesla and are now on a better road. It seems like things are slowly changing for the better. GM has the Ultium platform, Ford has the electric F-150, Mercedes is the first to get Level 3 autonomy (albeit in a very limited fashion). The times, they’re a changin’