imix sensors power bugs tracking issue

[alevy]

/cc @brghena

There are some un-diagnosed, rather serious interactions between RTS3 and the sensor power rail.

To replicate the problem, first install the Tock kernel from master or the bugs/imix-power branch of Tock:

$> cd ${TOCK_BASE}/boards/imix
$> make flash

The, install the imix sensors app:

$> cd ${TOCK_BASE}/boards/imix/apps/sensors
$> make flash

Finally, print the output from the app by running miniterm with DTR inactive (HIGH, attached to SAM4L reset) and RTS active (LOW, default):

$> miniterm2.py --dtr 0 --rts 1 /dev/ttyUSB0

You should get output resembling the following every second or so:

--- forcing DTR inactive
--- forcing RTS active
--- Miniterm on /dev/ttyUSB0  115200,8,N,1 ---
--- Quit: Ctrl+] | Menu: Ctrl+T | Help: Ctrl+T followed by Ctrl+H ---
Intensity: 281; Temperature: 23.64; Humidity: 19.60

However, if RTS is forced inactive (HIGH), there is no output:

$> miniterm2.py --dtr 0 --rts 0 /dev/ttyUSB0

Note that the console turns off hardware flow control entirely.

Finally, attaching a patch-wire between NRF_EN and GND (to force the NRF51 on) results in output all of a sudden.

More symptoms

Under mainline kernel, with PB[06] assigned to RTS3, the voltage on NRF_EN and SENSOR_EN are never quite 0V or 3.3V. Instead they are closer to 0.06V and 2.2V, respectively. Removing the assignment of PB[06] to RTS3 (by either leaving it disabled or enabling it as an input, e.g.) results in either 0.04V or 3.2V (which seems reasonable?). The overall problem is not fixed, though, and in fact, it makes --rts 1 stop working and the only way to get any output is to explicitly enable the NRF as well.

[brghena]

Finding: the I2C lines are screwed up while the nRF is power-gated because they are not isolated

Initial testing

Following the steps above on your branch (commit: efd845) with:

• PB[06].configure(Some(A));
• Power over USB from a high-current USB port
• miniterm.py on Ubuntu with --dtr 0
• nRF programmed with low power app
• Default state of !NRF_PWR_EN net

Running miniterm.py with --rts 1 and --rts 0 both result in the sensors app printing data correctly. I have checked that RTS actually changes values. I have tried non-high-current USB ports. Running off of a power supply I see that the board draws 13 mA (without USB attached). The !NRF_PWR_EN net is 3.3 V and the VCC_NRF_3V3 net is 2.8 V. The results are the same on both Imix boards at Michigan.

I cannot directly replicate your results with this setup. I find the RTS line to have no impact on the sensors app.

Testing nRF enable

In these tests, Imix is powered through USB (from a high-current USB port) and !NRF_POWER_EN is tied to different voltages.

Running the low power app on the nRF51822:

• Tied to GND: VCC_NRF_3V3 = 3.3 V
• Tied to 3v3: VCC_NRF_3V3 = 2.8 V
• Tied to 5v: VCC_NRF_3V3 = 0 V

Running the tock-nrf51822-serialization-sdk11-s130-uart-conn app on the nRF51822:

• Tied to GND: VCC_NRF_3V3 = 3.3 V
• Tied to 3v3: VCC_NRF_3V3 = 2.6 V
• Tied to 5v: VCC_NRF_3V3 = 2.6 V

Actually disabling the nRF

Noting that 2.6-2.8 volts is probably enough for the nRF to be running. I tried directly tying the VCC_NRF_3V3 net to ground. In this case, without the USB connected, power draw increases from 16 mA (running the tock-nrf51822-really-long-name-uart-conn app) to 24 mA. Moreover, while tied low, the sensors app does not function. Letting go starts it again.

Adding a "Starting" message to the start of the app shows that the app is not restarting. Rather, the SAM4L is hanging and then continuing the application. Inspection shows that the I2C bus connecting the SAM4L, nRF51822, and Sensors falls to 0.3 volts when the VCC_NRF_3V3 is zero. This causes the bus to hang.

The problem is that the I2C bus is pulled up to VCC_SENSE_3V3 but is connected to the SAM4L on VCC_MCU_3V3 and the nRF51822 on VCC_NRF_3V3. If you actually intend for the voltage domains to be separate and gate-able, the connections to the I2C bus must be isolated.

Things that are wrong

1. The transistors for power-gating components are not working. They are not capable of cutting off power from the nRF51822
2. The TWI2 bus is connected to three different voltage domains and isolated from none of them.

Fixes

We had to deal with both of these problems on the Signpost platform. Taking a look at the backplane hardware rev B (which works except for USB USB works fine tyvm -Pat) should be pretty informative in general.

1. We used the SIP32401A load switch to handle power gating modules
2. We used the FXMA2102 voltage translator/buffer/repeater/isolator designed for I2C applications

The SIP32401A has a really tiny package that's a little hard to get right by hand. If you expect to hand-solder boards you might want to find another part.

Also, both of these parts were selected to fit the low current draw requirements of the Signpost platform. There might be easier to use / less expensive versions that have a higher current draw.

[ppannuto]

Quick follow-up on those parts, the I2C isolator is actually the more challenging package. I had maybe a 75% success rate on hand-soldering those on first try, 90% on the power switch. That said, they're easy (ish) to validate install with a quick electrical test and usually one reflow was good enough to get them working so not the worst by any means.

[shaneleonard]

Following up with @brghena's post, I believe the NRF is getting back-powered through the IO lines from the SAM4L. @kwantam's explanation for this behavior:

For most i/o pins on CMOS chips the circuit that
drives the pin (i.e., that drives it high or low when the pin
is in output mode) is just an inverter. The p-channel device in
that inverter has drain-body and source-body diodes, and most
of the time the source and body are shorted to VDD. This means
that the drain-body diode goes from the pin to VDD ( pin -|>|-
vdd). Normally VDD is no less than the pin's voltage, so the diode
is reverse biased. But if VDD is at ground and the pin is at 3.3
volts, the body diode forward biases and charges VDD up to about
2.7-2.8 volts, which is just what you're seeing.

[shaneleonard]

I think this is essentially what you're also getting at with the I2C lines not being isolated, correct? This issue affects some of the other subsystems that we're trying to power gate, so I'm going to open a new issue ticket for it.

[shaneleonard]

The current proposed fix for this issue is to use analog switches acting as pass transistors. I need to isolate the RF233 SPI (4 pins) and two I2C domains (4 pins), so two of the 74LVC4066 should work (thanks @kwantam). If I use the SIP32401 to solve #12, then I can use the same enable signals to activate the switches. Using an analog switch might actually also provide a fix to #13 if we can selectively connect/disconnect the FTDI DTR line.

[shaneleonard]

This solution assumes that we are OK with manually pulling all relevant GPIO lines to ground when a module is not in use (eg the RF233 RESET/IRQ/SLP pins, etc).