Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hang in TwiMaster::Read #79

Closed
Avamander opened this issue Oct 4, 2020 · 9 comments
Closed

Hang in TwiMaster::Read #79

Avamander opened this issue Oct 4, 2020 · 9 comments
Labels
bug Something isn't working

Comments

@Avamander
Copy link
Collaborator

I just encountered a situation where the PT froze on line 91 of void TwiMaster::Read(uint8_t deviceAddress, uint8_t *buffer, size_t size, bool stop).

Line in question:

while(!twiBaseAddress->EVENTS_LASTRX && !twiBaseAddress->EVENTS_ERROR);

It probably should have a timeout counter of some sort because such a freeze is not good when someone has a sealed PineTime.

@Avamander
Copy link
Collaborator Author

tt_392 on Matrix said:

I ended up just waiting for the thing to lock up, then turn off twim, force the pins back in the right position, and turn twim on again

@JF002
Copy link
Collaborator

JF002 commented Oct 7, 2020

From our talk in the chat room, it looks like the TWI device is buggy and locks itself from time to time.
Adding a timeout in the driver and completely reset the TWI device when we detect a hard lock seems the only solution.

@TT-392
Copy link

TT-392 commented Oct 8, 2020

Here is how I ended up fixing it btw
https://github.com/TT-392/TT-time/blob/master/src/external/infinitime/i2c_pine.c

As far as I understood from the nordic forums (allthough I can't seem to find the specific thread anymore). It seems that you specifically have to forcefully reset your SCL and SDA pins.

@JF002
Copy link
Collaborator

JF002 commented Oct 8, 2020

@TT-392 Thanks for the fix! It looks like the TWI drivers hangs when SPI and TWI are heavily used at the same time... I guess you couldn't find another solution that would prevent the device from locking?

@rafacouto
Copy link
Contributor

Investigating this issue, I think it could be related to Nordic's hardware anomaly 109. Moreover, it could be also the same problem described on issue lupyuen/pinetime-rust-mynewt#24 affecting SPIM.

There are some workarounds in the mentioned document. Handling peripheral IRQ to wake up seems the best one in this case. See example code in 3.8.3 TWIM workaround subsection.

@JF002
Copy link
Collaborator

JF002 commented Oct 22, 2020

I've just had a look at the anomaly referenced by @rafacouto. This is pretty bad : a race condition inside the CPU. And the workaround are quite complex to put in place.

However, the anomaly specifies that SPIM TX and TWIM TX are affected by the anomaly. TWIM RX is not referenced. But RX could be locked because TX caused the anomaly just before?

Anyway, I'm trying to reproduce it so that I can analyze that issue further... with no result for now
I've tried using InfiniPaint (paint app) from develop, and the pong game developed by @ColdBrewCaffine (https://github.com/ColdBrewCaffine/Pinetime/tree/Run-dev-2).

Soooo... can you help me find a way to reproduce this issue more or less reliably?

@JF002
Copy link
Collaborator

JF002 commented Oct 23, 2020

Ok, I can reproduce it! I just have to wait longer.
The fix from @TT-392 seems to work, even though I would have preferred to be able to actually fix the issue instead of using this kind of workaround.

For info, the 5000 loop iterations correspond to ~160550 CPU cycles, which corresponds to ~2.5ms @64mhz.

Now, I need to check that the code behaves correctly when the twi transaction is aborted.

JF002 added a commit that referenced this issue Oct 23, 2020
When the driver is stuck in an infinite loop for more than ~2.5ms, the TWI device is re-init and the transaction is retried.

Read() and Write() return an error code.
@JF002
Copy link
Collaborator

JF002 commented Oct 23, 2020

I've implemented the fix recommended by @TT-392, it's running fine for more than 4 hours with the pong game developed by Electrolyte.
See this branch : https://github.com/JF002/Pinetime/tree/fix-twi-hang and this PR : #111

@JF002
Copy link
Collaborator

JF002 commented Oct 27, 2020

Fixed in #111

@JF002 JF002 closed this as completed Oct 27, 2020
tgc-dk pushed a commit to tgc-dk/InfiniTime that referenced this issue Nov 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

4 participants