Sorry I haven’t posted more. I have done some cool projects, but finding time to post has been hard. To help fill that gap, here’s a report I wrote up detailed a small chapter in my involvement with Team Blue Devil Ocean Engineering, which is Duke’s entry into the Ocean Discovery XPRIZE, a world-wide contest to develop the technology to map 500 km^2 of ocean floor in 24 hours. It presupposes some knowledge about the project, which you can find in this brochure or even these slides, or you can just dive in and have fun gawking at this crazy thing we built, sunk 2km deep in the ocean, retrieved, then debugged.
Details of an intense 48-hour effort to build an deep-ocean-survivable Arduino control circuit for underwater rockets is after the break. It was written as an after-action report for the project, so the language is a bit drier than usual, but I think it’s still a fun read.
Original control system issue
After epoxying and construction of the rocket test pod by the mechanical engineering student group, it was found that the hall effect sensor and data lines of the USB connection for the control system were both non-functional (though USB power worked). For the hall effect sensor, the likely cause was the use of dupont connectors to hold the fine leads of the sensor rather than solder. For the USB, it’s less clear, as the connector was a spring-loaded microUSB, which has worked in the past. Current hypothesis was that it was because the cable was torqued at an angle, reducing contact. The result of this was that the control system had no way to be triggered (hall sensor) or reprogrammed (USB connection), and thus was a bust.
Recommendation: Solder all internal components. Where connectors are needed (e.g. USB), use zip ties or similar to mechanically secure the cable in a low-strain orientation.
Replacement control system
To meet a rapidly approaching test window on the ocean, I constructed a new control unit to be epoxied on top of the pod. The schematic is below:
The control circuit was based on an Arduino Nano, IRLB8721pbf MOSFETS, and four 18650 battery cells in series. It was soldered together on simple perf-board using solder bridging. To fit all the MOSFETs, a highly compact MOSFET+LED layout was used:
The controls were simple a button and LED on another protoboard. When finished, all wires were mechanically attached to the board with zip ties.
Recommendation: Future systems of this nature should use a custom PCB to greatly simplify construction and improve reliability.
Firmware and power draw
Firmware was developed to allow deep, low-power sleep of the controller, with an interface to begin/abort countdown based on a single LED and button. Code is here:
While in sleep state, if the button is pressed, the LED fades on. If the button is held at this time, it begins a slow blink to indicate that you’re enabling the countdown. If held long enough, the blinking becomes rapid, indicating that the countdown is active. After this point, the LED blinks once per second, with the duration of the blink proportional to how close to activation the timer is. It starts at 0.1s on when no time has elapsed to 1.0s on (100% duty cycle) when the entire countdown is done and activation is imminent.
The countdown can be aborted by holding the button for 5 seconds, at which time the LED will fade out, indicating a return to the sleep state.
If the countdown gets to zero, the rockets are fired one at a time, 8 seconds of activation per rocket, with a 2 second delay between each one. That means that total activation takes 6*(8+2) = 60 seconds. After a completed activation, the indicator LED is left on when the controller goes to sleep as an indicator that a firing attempt has occurred.
Current draw for the system was around 5mA during sleep mode and 25mA during active mode. The indicator LED consumes an additional 8mA when lit. If we conservatively assume a 2000 mAh battery, that’s 2000/5 = 400 hours = 16 days of sleep time.
Deploying the new control system
The new control circuits were placed in a new cylinder atop the pod, then soldered to the rockets igniters, which were installed and sealed in wax as per usual.
The new controller was then epoxied. This consisted of pouring/curing a small amount to act as the floor, adding the circuit, then pouring the rest. During this process, the holding tube was damaged, and so duct tape was deployed to allow enough epoxy to cover the circuit.
Unfortunately, during this process, the battery failed. We observed that the plastic case that held the 18650 cells had warped, likely due to a combination of the battery spring tension and the plastic becoming soft at the high temperature at which the epoxy set. We were able to recover this by adding an additional battery pack attached to the charging line, which still connected directly to the V_IN of the controller. We made three attempts to build an epoxied battery pack: one using modular one-cell 18650 holders (zip tied to prevent warping at temperature), one using a pair of molded 2-cell 18650 holders (zip tied to prevent warping at temperature), and one using directly soldered batteries taped together. Only the latter pack worked: both of the other packs succumbed to the same warping and disconnection as the original.
Recommendation: do not use spring-backed plastic for anything in epoxy, especially plastic battery packs! We have since ordered a tack welder to better construct custom battery packs, as using traditional soldering techniques on batteries is not recommended, as the bond isn’t great, and heating lithium cells to that temperature is generally unwise.
The add-on battery pack was soldered to the control system’s charging port and sandwiched under a piece of plastic held in compression with nuts on the threaded rods of the pod. The control panel was impaled on one of the rods and compressed as well. The rods and nuts are aluminum to survive ocean corrosion.
The pod was deployed to a depth of 2000m in the ocean on a fishing line. Upon retrieval, the electronics appeared non-responsive, and no rockets had appeared to fire. Because of the indeterminate state of the system and concerns over rocket ignition while onboard the boat, the battery wire was snipped.
On return to Duke, the pod was examined. It was physically in good condition, with only one small part of the 3D-printed exterior damaged (which is fine). The battery was in good condition (>16V). The electronics appeared to power on normally, both through USB (5V only) and V_IN (the battery line). The interface button was corroded and didn’t work at first, but a few heavy presses to break up salt and corrosion restored it to working order. The LED faded on and off as per usual.
However, three key misbehaviors were identified:
- Very strange behavior was observed in the LEDs onboard the Arduino Nano. The blue power LED did not light except when the pin 13 indicator LED was on. Pin 13 goes to both a green onboard LED as well as the white LED on the interface board. The reference schematics for the Arduino Nano was consulted (as it is open hardware), and we verified that the circuit for the power LED is literally: 5V -> LED -> Resistor -> Ground. We were not able to find any satisfying hypothesis for how the power LED could appear to be tied the behavior of an LED on an IO line.
- While the device could be powered via USB, it did not appear able to communicate over USB. Windows made no attempt to detect it (no sound, refresh of the device manager, or log event from USB diagnostic tools). While powered from V_IN, connecting USB would sometimes reboot the Arduino (it always should); no consistency was found to when this would happen.
- Late in testing, after the device had been left powered by V_IN for a while, the indicator LED started flashing rapidly but intermittently with no discernable pattern. Because of the fear of detonation, power was cut, and current-limited USB power was supplied instead. Before long, a similar blinking occurred. Eventually it stabilized, and the LED did the 4-blink pattern programmed into the firmware on boot. Because this LED lights briefly during the instant of a reboot (a firmware effect), we believe that something was causing the Arduino’s reset link to be triggered intermittently.
Physically, we observed that the exposed USB cable was filled with seawater, as when the cable was cut so the corroded connector could be replaced, moisture was encountered, and later a drying of the skin consistent with seawater evaporation was felt on my hands.
Based on the above, we believe that seawater traveled down the jacket of the exposed USB cable and entered the epoxy at the Arduino Nano’s USB port. There, it seems to have corroded or otherwise interacted with the board, most likely causing indeterminate behavior in the FT232 USB/serial chip onboard. This chip is connected to the ATMEGA’s reset line and is also responsible for USB negotiation an attached computer, both of which malfunctioned as documented above.
Recommendation: Do not use jacketed cables protruding from epoxy. Instead, strip to inner insulation, or even better, bare metal inside the epoxy (perhaps treated with an anti-corrosion spray). This should hopefully as a moisture barrier to prevent pressurized water from reaching and corroding the electronics.
- Solder all internal components. Where connectors are needed (e.g. USB), use zip ties or similar to mechanically secure the cable in a low-strain orientation.
- Given more time, a custom PCB is recommended for control rather than ad-hoc construction.
- Battery packs should be constructed using soldering or tack welding techniques; spring loaded battery holders (and load-bearing plastic components in general) are ineffective unless special steps are taken to reduce the epoxy curing temperature (e.g. alternative low-temp epoxy products, including pre-cured epoxy chunks to reduce reaction volume, etc.).
- Avoid having a jacketed wire exit epoxy. Instead, strip to individual insulations or even down to bare conductors inside the epoxy to serve as a moisture barrier. Even better, don’t have any wires exit the epoxy at all if possible.
Physically, the pod and its rockets are in great condition. We showed that the design can survive down to a depth of 2km. I believe that the rockets would likely ignite if triggered at this time.
Because of the indeterminate behavior of the damaged control system, a new control system will need to be added if the pod is to be successfully tested in the future. The procedure in this document is modified to incorporate the lessons above, it should be possible to produce a viable control system. Also, the battery built for this test can be reused.