I was really quite energized this week and had planned quite a show for the weekend. But we were kind of overwhelmed with orders and order problems towards the end of the week, and so increasingly I’m finding that the only time I can put focused effort into a project is at home in the mornings or on the weekends at the shop when my “help” isn’t there to “help”. That’s tradiionally when I have edited and uploaded videos.
But I’ve been obsessing recently on the Tesla Drive Unit. Collin Kidder and I have been going back and forth in kind of intense fashion between a controller box and the EVIC display to control the Tesla Model S drive train. And actually, we are getting very close.
So close that I went to the shop Saturday afternoon with some new code we’d been working on intending to shoot a video segment showing this remarkable display we have now that controls the drive unit.
As always, it didn’t work “exactly” the way I pictured it from the comfort of the bedroom. But to give you some idea, we’re kind of doing to doing a 500 slot ring buffer to do a continuous averaging of watt-hours per mile smoothly so we can predict the range remaining in the pack in miles per hour and kilometers per hour. This adds a new sample about every 10 seconds and averages it over the last 500 samples. It won’t be any better than the almost comical one in the ICE Escalade, but it’s traditional to have this displayed.
So I kept at it. Deadline 6:00 PM Saturday as I had promised the wife I would join her for a bridge games with the Matthews.
I am pleased to report that the look and feel of the screen are pretty much what I want and pretty much at least resembles the instrumentation in my Model S. It’s all functional and has pretty good response.
So I get a glass of tea and light up a Camel. It’s 5:43 on Saturday evening and I haven’t shot a second of video as yet. But I do have it calculating a plausible remaining range, and doing the state of charge thing and the amphours used and kilowatts used and it is all hanging together. I can put it in any gear from EITHER the PowerkeyPro 2400 panel or the touchscreen. Creep on/off demonstrably works. Regen increase and decrease. My trip meter works. The odometer resets. The SOC resets at a touch. And I finally have the colors right on the very strange logarithmic scale Tesla uses to show power output or input on the right side of the dial. Life is good. But I’m going to have to shut down and go to a bridge game.
POP. BANG.
What was that? Sounded like somebody banging on one of the closed garage doors. BANG POP POP. What the hell?
I went into the next room to see who was banging on the door, but as I reached to open the door, the POP POP BANG sounded again but BEHIND me. I walked over to the Better Place battery pack from the Renault Influenza that we use on the OEM components test bench. BANG POP POP. These are actually pretty loud. What the……?????
This pack was right out of the cargo container and we never even attempted a bottom balance. We were only going to use it for testing chargers and DC-DC converters and the UQM test bench. But as the result of one of our assclowns playing around with the bench while I wasn’t in the shop, it had drained down very slowly overnight to a very low level.
It seemed to charge back up ok. But never quite got to full charge. So I had hooked it up earlier in the afternoon to bring it up some more.
I quickly shut off the charger and cut off the contactors. But it continued to BANG and POP irregularly. I can’t leave to go to bridge with it like this I’m thinking. As there had been several of these “no show” moments in the past few weeks where I threw my wife under the bus with regards to one plan or another, this was not really good. I can’t believe I’m doing this again.
Suddenly the pack begins to issue the familiar white smoke – just a bit at first, then more. The pack weighs 450 lbs, and the fork lift is at the other end of the building. I went over to the wall water spigot, glad I had a couple hundred feet of hose there to water our grass. No hose. Assclown somewhere had made command level decision to move it down to the basement in the other building apparently. There was a hose, but it was four feet long.
By this time the white smoke was coming out pretty good. I don’t know why, but I was curious what the temps were. So I grabbed an infrared gun and shot all the cells. Most were warmish in the 35-40C range but there were two sitting at 95C. Not good.
Suddenly the pack spewed a spear of sparks and flame about six feet straight out the front – right where I had been a moment before. And then it exploded into a massive fireball shooting flames up to the ceiling with such velocity that they splashed laterally from there.
It would have made excellent video I must say. At this point, I’m out of altitude, airspeed, and ideas all at the same time. I jumped in the Tesla and backed it out of the building. For some reason, I closed teh garage door after I was out, as I always do. I then had a positively exasperating wrestling match with the car for control of my cell phone to dial 911. I mean this car wanted to do ANYTHING except dial the phone. It played some Kriss Christopherson “Help Me Make it through the Night” (right) and some Rod Stewart. But it must have taken me three or four minutes to get it to dial 3 digits.
Cape Girardeau actually has an excellent fire department and they have a station about three blocks away. They were there within five minutes. I opened teh garage door to let them in and it seemed the flames had died off – I guess I had inadvertently shut off the oxygen. But black billowing clouds of smoke poured out of the building.
They quickly got it under control. But the way Fire Departments work to put out fires, they mostly disassemble the building. They cut power lines, climb up on the roof, open it up, and of course dump several shiploads of water down into the space. It looked like they were cutting the cables on the roof connecting the solar system, the air conditioning condensor, and anything else in sight. The $30,000 roof I had just installed two years ago was torn up and opened up with chain saws. Yeah, they’re that good.
What did I do? Well, I answered a few questions from the fire department investigator but that was pretty short when he found out I don’t HAVE any insurance. We have a lot of buildings around here go up in flames with a LOT of insurance on them. I don’t carry any. So I’m pretty much off the prospective arson prosecution list – shortening the interview rather quickly.
So I went home to play bridge. Brian Perry kind of hung out to try to secure the building in the aftermath. But I went home and played bridge.
Not very well I fear. We got clobbered. I guess my head wasn’t into the game but the wife, as almost always, was very understanding of my poor bidding and play.
It’s Sunday morning. I guess I’ll go down and see what’s left. But we have no roof in rainy November. And no electricity at all – they pulled the meters of course. My OEM components test bench is no doubt gone and I actually had a special bench spruced up on one side with all the various connectors (rather expensive connectors) needed to use these components. I had spent over a year tracking all that down. No hope. It was too close to the battery pack.
I firmly believe that all happens for a reason and ultimately for the good. But this one will be interesting to see play out.
It’s a very dark, though sunny Sunday morning.
Jack,
Sorry to hear about the fire and just as a precaution, think I will go out in the garage and double check the voltage on the Better Place module that is sitting in the front of my garage. I knew lithium batteries could gas and burn but from seeing the battery destruction demonstration at the airport a couple years ago, did not realize it would be that violent. I guess the Better Place chemistry is different than the LiFePo4, so guess I am going to need to keep a close eye on the modules in my Fiero and the one still in the garage.
If you need a hand cleaning up, maybe some of us could lend a hand. I have some vacation left this year or would be willing to come out on a weekend.
Randy
Since I got one of the better place packs – I’ll keep an ear out for POP-BANG!
This may slow some of your projects down a bit, but they will be better for the upgrades you do while building them again.
No one was hurt, the equipment can be replaced, the building fixed. AND it’s yet ANOTHER interesting story. An EXPENSIVE story to be sure, but very interesting. I’m sure that the post-mortem on the battery pack will expose more interesting elements. That is my hope since the presented facts are somewhat disturbing
Hang in there – we’re still behind you.
Wow, that really, really sucks. I’m glad to hear that you made it out OK.
ME? Thanks but I’m quite invulnerable except when playing bridge. I received a special dispensation from the Pope at age eight. I can’t actually be physically harmed.
I have described what happens in these overcharging lithium battery fires actually quite accurately in the past. But no one is ever there to SEE them and see the initial violence. I actually DID in this case. I observed the entire process. It explains the total destruction.
Picture if I HADN’T been there and/or the fire department wasn’t literally 3 blocks away. You’ve all seen the shop. The ceiling is what? 16 feet? 18 feet? Raw wood with steel girders. The flames were SPLASHING over the ceiling. EVTV would have probably been TOTALLY destroyed, burned the ground with everything in it.
As it is, it’s kind of a mess. I hear the front of the Mini Cooper is pretty messed up. I probably lost a bunch of connectors it was hard to collect. And we had a stupid number of DC0-DC converters and chargers stacked up there right next to the battery pack. I’m sure it cost me a laptop.
But I’m told the TEsla test bench and all the stock is actually in good order. We had JUST taken delivery of 94 additional DMOC 645 inverter controllers for example. But we’d moved them to the battery lab back building the same day, thank goodness.
Day in the life. But yes, it sucketh with great enthusiasm.
I as well am willing to make a long weekend trip to help you clean and rebuild.
Maybe the fire department should investigate further for their own education. I’m confident they see a lot of ICE vehicle fires and are familiar with them, but likely don’t see many EV battery fires. A battery acting as both an ignition source and accelerant might be educational to them.
Was the battery charging or discharging, or was the original bms still hooked up?
Maybe a video of placing a Nissan cell on a 12V charger out in an empty parking lot might be educational and good comparison to a CALB cell.
I’m very grateful that nobody got injured. Had you went to the fork truck before the water hose, you would have likely been injured sitting in the truck so close to the battery.
Had the shop cam kept an 8 hour history, you would have had video of the whole incident until the fire dept cut the power…
Jack,
I am sorry this happened to you and very happy that no one was hurt. A lifetime ago I did some volunteer firefighting and the required bit for the Navy as well. I think by shutting the door you saved the building.
Jack I am so very sorry this happened my prayers are with you at this time
oh shit!
That’s EXACTLY what I said….
Jack Rickard
Ouch!
Sorry Jack
Let me know if there’s something I can do to help with the cleanup.
Battery overcharging, yep, you’ve covered this topic extensively and on more than one occasion. Also these are different cells than a lot of us are using; I like my LiFe cells more and more all the time.
With OEM cells of different chemistries becoming more available and at Cheeper prices they look tempting…times gone by, don’t hear a lot of battery/car fire stories…cells must be OK, right?.
Why not buy a used Volt or Leaf pack for $2500,$4000 respectively. Have we been lulled into a false sence of security pertaining to these cells? Are they really like our LiFe cells and fairly stable as the lack of fire stories would make them seem?
Or is ‘Buyer beware’ still the appropriate approach for these other chemistries?
Maybe a BMS nanny is needed for some of the chemistries that lean more toward the incendiary device side of the battery equation. I have no idea, but until we fully ‘test’ all the various chemistries, assuming that they can all be treated the same, packaged into the battery box and forget about them, is a tenuous proposition at best.
Whatever else comes of this setback, sales of your excellent CALB LiFe batteries may be on the rise!
And thanks for the overcharge test on these cells…just really sorry it wasn’t a planned event.
I guess I think a BMS is probably in order with these. I’ve had little luck bottom balancing them.
The issue is that if you loose a cell, it ups the voltage on the others, and they are 2S2P WITHIN the can. If one of the four go out, it will take the others.
Our CALB strategy is much simpler. The same applies. If a cell goes bad and you still fully charge it, the same thing will happen eventually. But we tend to kind of unddercharge anyway, they are hardier, and by bottom balancing, we just don’t seem to lose the individual cells.
But yes, this was something to watch. The bad part was that this went on or 15 minutes and I was unable to think of a single thing I could do to stop it. No water. It weighs a ton. Not on wheels.
We’re going to put packs on wheels in teh future and with someway of quickly disconnecting them and wheeling them out into the street.
Jack Rickard
Expensive lesson, Jack.
But you are one of the privileged seeing and surviving this. Take it as yet another birthday with fireworks and all.
Happy Birthday Jack.
I am glad your are still with us.
You have got 4 arcwelders competing to ignite a bundle of aluminum. I dont know if a garden hose would have been a good idea at that very moment. Burning aluminum and water gets you hydrogen. Those 5 long minutes may have been enough time for the aluminum to burn and leave the water alone.
It is you who has seen. Please excuse if I have said anything silly.
Peter Dambier
“I guess I think a BMS is probably in order with these….”
Maybe using a different supplier than Boeing who had two (maybe three) of these light off in electronics bays despite the use of duplicated BMSs
Holy crap. I’m glad you’re OK Jack, but … holy crap.
sorry ,just meant : don’t they have special switches on the outside of the building for the purpose of disconnecting power? it’s sort of required right? so why the mindless destruction?
so sorry to hear this news.Glad you’re safe and that’s all that matters…
Hi Jack
Sorry to hear what happened!
Is there anywhere that I can send a donation (no strings attached) that you can use to rebuild the shop, re purchase the lost components, but some food for the people coming to help you, maybe a gift for your wife (was I tequila that she liked?) or maybe some wine or beer for you?
Please let me know if that is possible.
Thank you for EVTV.
Regards
Carsten
Is a shop rebuild fire sale in the works? 1% off?
Jack,
As others have said I would be more than willing to come down and help with the cleanup and or setup of gear to replace the lost benches. I know now is not when you want just anyone poking around the shop so perhaps a few members of the hack team and a few other trusted members could come down. I have always wanted to go on the roof of EVTV so perhaps we could put the solar back together, just an idea. If you want me to set something up, let me know. I also have the next presentation from EVCON ready to upload. I just need a day to upload it. It is the session with Nick Smith and Steve West, perhaps it can make up for the lost video session this week.
Im sorry for the loss of the gear, and part of the roof. But i’m glad that you are ok.
Byron I
First and foremost I am glad you are OK. As to everything else, I am sure your needing this as an excuse to rebuild the shop bigger and better then before! Bluntly, I am actually surprised alcohol was not factor! Keep on trucking Jack!
All the best,
Aaron Lephart
That’s absolutely Dreadful!
That’s scary! Glad you were there to catch it and that you made it out of the building.
I feel your pain. I personally burned my own shop down on july 20 2012. Mine was my own fault in an incident involving a cut off wheel too close to a gas tank and the results were much more dramatic. flames 40′ high, burnt to the ground. On the bright side I made several improvements when rebuilding, the new shop is much more efficient but I can still feel that moment in my gut. Glad you’re ok. Sorry we didn’t make it for Not EVCCON, we cancelled our trip when it was cancelled, fools we are. We do also have first hand experience rebuilding so if you’re short on hands, let me know, I’ll talk the old man into a road trip
That sucks.
I’m very glad no one was injured, that special dispensation from the Pope must be real handy!
I’m very sorry for your loss and wish you the best in rebuilding.
-Otmar
If there is anything useful I can do from 3000 miles away, I’d be happy to do it.
When the dust settles and you have time from more urgent things, I’d love to have as complete a description as you can manage of what was done to the pack leading up to the fire (approximte, voltages, currents times etc.)
I have had good success bottom balancing lithium manganese and cycling without balancing but they rebound much more than LiFe and the knee is much more vicious. Also we have observed a significant number of cell failures across all chemistries.
Jack,
dont giveup on that laptop. Dry and clean the hard disk and put it into an usb case.
There might be intersting data on it.
My God !
Jack I am glad you are OK… Thank God ! I’ll keep up my prayers for you !
Gambatte Kudasai !
Hi Jack
I am truly sorry for your loss.
I am afraid that it isn’t the fire department that caused worst damage.
In my experience the acid fumes from fire will be a bastard and you have a lot of cleaning to do.
It also look like you have firm support.
I wish that I can do something for you from the other side the pond.
But I guess, if you reinstall the donation button a lot of donation will come to you.
it might come in handy, if we are enough people, and I’m sure we are !
I hope the best for you and I’m glad that you are OK.
Best regards Allan
Ouch! You warned us at length of these battery fires, but to have one take out your own shop to confirm what you already knew… Having batteries on wheels makes sense, if there’s someone around who dares wheel one outside while it’s popping, banging and spewing flame. Maybe a segmented and well vented battery bunker with sprinklers. A repurposed shipping container or the like that’s already outside, with cables going through the walls and into your shop.
I always think of all the power that’s held in a fully charged lithium cell and what would happen if it were all released in a short period of time.
Well it maybe of no consolation but your actions saved a lot of hard earned EV ground that could have been reduced to ash. Count me in, if you require some assistance for some reconstruction or clean-up. KCGI is still in the GPS routing.
Holy crap! Glad you’re okay.
I like the idea of quick disconnects and battery packs on wheels so you can jettison them quickly. Might be good to consider using a temperature controlled relay to cut the charger off when the temp probe heats up.
Now you’ve got me thinking I’m going to add some temp safety gear to my EV.
http://www.ebay.com/bhp/temperature-relay
or
http://www.amazon.com/Lerway-All-Purpose-Temperature-Controller-Thermostat/dp/B008KVCPH2
Or program an Arduino to look at both high temp threshold and rate of temp change.
Glad everyone is safe.
Dale
Thanks Dale,
I’m working on heating ideas so I can heat my pack for operation and charging during these next few colder months. Always nice to have links to items handed to me on the day I need them. I’m actually running a heating experiment this instant measuring temperature rise over time vs. heating input… It’s in the workshop, I’m in the house…time for me to go check and make sure nothings on fire yet!
BTW, anyone have input on recommended temperature range for Chevy Volt batteries?
Preliminary research is telling me this, but this is indirect and not from perfectly reliable sources:
http://gm-volt.com/forum/showthread.php?5243-Volt-thermal-management-system-temperature-band/page13
http://www.nrel.gov/docs/fy13osti/58145.pdf
Operating Temperature Range:
Ideal: 73-89 deg F (23 C – 32 C)
Reduced Range: 33F – 73F (1C-23C)
Reduced Cycle Life: 90F-100F
Damaging/Don’t Do it: Under 33F and Over 100F
Charging Temperature Range:
Ideal: 73-89 deg F (23 C – 32 C)
Reduced Cycle Life: 60-73F, 89F-100 F
Damaging/Don’t do it: Under 60F and over 100F
There’s probably an EVTV video covering this subject, I should go searching for…
Here is at least one EVTV episode on the heating pad subject: https://www.youtube.com/watch?v=kHN5PhaUD7U
1:30:00 – 1:50:00
And EVTV sells the heating pads I need: http://store.evtv.me/proddetail.php?prod=batheater&cat=16
Jack;
Sorry to hear about the fire, but glad that you were not hurt. Only ~ 3 hours away, so please contact me for any help that I can provide.
Jack,
Sorry to hear. Glad you are OK. That was a close call.
I’ve been getting more and more anxious lately working on my own larger packs. They are so silent and seemingly inert, but they are just very patiently waiting for something to release them!
Thanks to the internet and your openness to sharing, your lessons are our lessons; if we listen and pay attention.
Thank you for your continued support of the EV early adopter club.
Kraig
Sad 🙁 Glad you’re okay, people are more important than stuff, even amazing stuff…
What a bummer…
That Better Place pack has been my favorite bench pack, but it clearly needs to be well tended.
If that is the pack that is the “charger/heater/DC-DC converter bench” then it was fine and well last month.
Put me down as a helper for shop clean up and rebuild too.
But in the end, glad to hear you and yours are safe!
Last night I went out to the shop and built a wheeled cart and set my Better Place test pack onto it.Now I have an easier way to “jettison the warp core” if needed.
As violent and horrific as it was, I’m sure 20 gallons of gasoline in a tank catching fire would have caused much more damage.
Gotta get a battery to power up that NetCam and point it in the right direction.
And get that drone flying again.
It is not our curiosity – of coarse it is, but I guess it is a good idea to see what the remains look and to get roof, heating, drying, ventilation and that public charging …
Get a bistro on the roof to keep your workers happy and to invite your neighbours as soon as the biggest mess is over. Tell them how lucky they are. Imagine a tank of gasoline had exploded with all those gas filled cars.
Just a little fertilizer making a lot of black smoke.
Yes we are missing the shop cam and I am afraid it might come back with a differerent ip address.
I’m here if you need me.
Is hosing a LiPo Battery OK ? What kind of Fire Extinguisher should we have nearby?
Perhaps Elon knows what he is designing after all using individually fused LiPo cells no larger than 18 x 65 mm. ( 69mm protected ) Anyone know for sure if the 18650 cells in the Tesla 24V Bricks have the protection chip? If so .. as far as the main BMS is concerned it’s just looking at 6S (24V) segments ??? Beauty in Scaling. . Will be interesting to see the OCP layout of the Powerwall. What happens if Billy backs the F150 into one? Byebye house? Underground Battery boxes looking pretty. Me thinks selling A123 to a foreign firm was a National disaster. We owe you Jack – you may save some lives.
Of Interest – FWIW – I was re-charging a 2015 NEW 8S 66Ah Leaf LG Pancake Stack balancing to 3.1 V and back up to 4.0V. Icharger 4010 @ 4A , less than 1 mV between cells, and it setoff a fire alarm upstairs 2 rooms away.. All Normal, Cells cool. Likely the cells emitted ions that trigger smoke alarms. Don’t know 4 sure.
Huge commiserations Jack. 🙁
My knowledge of the operational intellect by fire departments in every case has been very poor. An example story follows…
.
I was the only operational fire team guy at work when the oxygen meters went off. We used liquid nitrogen with the circuit board manufacture. They arrived. We rigged up with tanks They were utterly clueless what the meter was in my hand. Could not stop them wandering off looking at the ceiling for smoke. The only reason for calling extra hands in case someone needed dragging out. (me).
Most lads are in it for impressing the chicks that do not have phds.
.
My Nissan Leaf never lets any cell charge over 4.11V. That’s what needs hacking. 🙁
That sucks to read about, and most definetly worse to live through. I had 2 thundersky cells pop, inches from my face as I checked other cells while charging. It scared the shit out of me, and damn those little fuckers make tons of smoke. Glad to hear you’re alright Jack, you have a committed group of people who’ll do whatever they can to help
Like so many others here I’m sorry for your loss, and I’m glad it wasn’t a bigger accident. I hope you (as an intelligent person you are) see this as a turning point, a chance to rethink the operation and re-visit your safety standards now that you’ve seen with your own eyes the energy of 24 kWh of battery turned into an incendiary device. Separate battery dedicated area with fire proofing glass doors and fire proofed ceiling/walls (possibility to seal the space from getting oxygen?), “only lifepo4 for test benches” policy, a written plan called “In case of fire”, separate fire hose units (or whatever is needed to put out a lithium or other fire) etc..
Anyhow, wish you all the best and lots of strength (physical and mental) for the repair process to come.
I sounds like more than one cell caught fire… which I never expected with the LiMn2O4 chemistry. They typically don’t go nuclear until around 485 F — which is barely 35 degrees less than LiFePO4. I have thrown lipo’s directly into a fire, and have never really seen much action out of them.
Does this mean that if a LiMn2O4 starts to burn, it will more than likely set off all the adjacent cells?
Throwing a battery into a fire is a bit different than a battery pack catching fire. With a charged battery pack, you not only have the raw materials as fuel but all of the stored electrical energy. When you have multiple series / parallel cells, as the pack starts to fail, you can have short circuit paths and lots of arcing. I had a 20KWH LEAD acid pack develop a short with some battery cables. The short was several feet from the batteries themselves so the batteries survived. However when the smoke cleared (after the batteries were depleted), there was a 2.5″ diameter hole through a 1/2″ steel plate where it arced to the plate. Steel melts at >2500F, aluminum at 1200F.
Jack, I’m glad to hear you’re safe and sound in Cape Girardeau. From the paucity of flaming LEAF reports I’d started to think that the chemistry was non-perturbable. A reminder that when storing that much energy, the genie has to be let out slowly regardless of how purdy the bottle is.
OT? Yehu has got a problem.
u-tube “The Day a Cop tried to steal My Samba”
No comment about ideosyncrasies comparing Roman Law as in Italy or Germany or the law in english speaking countries …
Except in Germany Rache (vendeta, revenge) and Recht (law) are very much the same spelling and pronunciation. It is as in a regatta or tournement after a game the loser gets a revenge or second chance in a new game. But cops are not always good sports. In the UK they are but that is why they are called Bobby not cop.
Coincidence Jack and Jehu spending time with read tape right at the same time?
Wow so sorry to hear that Jack. I really feel for you. You are displaying a great attitude.
IIRC Jack indicated this pack had been taken to the bottom through small continuous parasitic currents. I suspect the original applications of these cells prevented this and activated lock-out fault codes. If this occurred in the OEM applications I suspect the vehicle fault codes being set may have locked out charging until cleared by the dealer. My BP pack has the air vent on one side but no air gap whatsoever between the cans. The BP pack allows direct contact between adjacent modules where as Tesla spaced the cells enough to eliminate direct-contact conductive heat transfer between adjacent cells. Coming soon to an EVCCON near you will be innovations to mitigate the risks……..but then again I could easily be “too tricky by half”. I dislike building a pack like the original pile of graphite on the gym floor in Chicago with nuke-grade uranium inside.
Stanley A. Cloyd
20 lb Pro 20 carbon dioxide fire extinguisher ?? $300-400. Less clean up.
The decay from thermal runaway provides its own Oxygen, iirc. The strategy for extinguishing is to apply massive cooling, which in practical terms means flooding with large amounts of water.
does this mean there will be a fire sale???
Pingback: Main battery - 1st vs 2nd design battery VIN - Page 2 - Chevy Bolt EV Forum