OTA Crashing WeMos device #530

sillyfrog · 2018-06-24T06:00:48Z

After much testing, I have been able to find one of the examples is able to replicate my issue. The most basic example as per http://marvinroger.github.io/homie-esp8266/docs/2.0.0/quickstart/getting-started/ does work OK. The TemperatureSensor does not. I think it maybe related to having two temperatureNode.advertise lines - but no idea how.

I'm using a WeMos Mini Pro v1.1.0 and a v1.0.0 (both have the same behavior - so fairly sure it's not the device). I'm sure OTA used to work on these same devices but that may have been Homie <= 1.5. I'm only recently getting back into OTA updates using the v2.0 firmware (latest stable release).

Basically, if I upload the "TemperatureSensor" example, then change the version to v1.0.1 (say), and recompile, and try to upload the newly built firmware using ota_updater.py, I get an Exception right away. The decoded exception looks like this: https://gist.github.com/sillyfrog/1e66895dd9b3ed576d31514c7bcface2

I have tried using both Arduino, and PlatformIO (each with their own checkout), and ensured all libraries are up to date.

Things appear to otherwise work OK, until you go to do an OTA update, and then things fall apart.

Happy to provide any more information you may need - and I feel I'm just doing something silly, but not sure where to start looking :/

Thanks in advance.

Cheers.

The text was updated successfully, but these errors were encountered:

timpur · 2018-06-24T07:10:54Z

Currently looking into another ticket, will get to this asap

sillyfrog · 2018-06-25T10:54:24Z

Thanks! If you have any thoughts as to where I could start looking, happy to take a crack and start poking around - it's pretty weird! :/

CWempe · 2018-09-19T15:09:50Z

Same here.

I finally wanted to start using OTA, but I cannot get it work.

When I send a firmware via homie-ota server I get a reset of my Wemos D1 mini pro.

eceiving OTA payload
↕ OTA started
Triggering OTA_STARTED event...
Firmware is base64-encoded

Exception (3):
epc1=0x4024162d epc2=0x00000000 epc3=0x00000000 excvaddr=0x40031e61 depc=0x00000000

ctx: sys 
sp: 3ffffa50 end: 3fffffb0 offset: 01a0

>>>stack>>>
3ffffbf0:  4022e0d0 3fff40a4 00000000 4022e079  
3ffffc00:  00000000 00000003 00000001 00000002  
3ffffc10:  3fff251c 00000000 00000000 3fff4744  
3ffffc20:  3fff3cb4 00000000 000001e1 00000003  
3ffffc30:  00000010 3ffffcd0 3ffffcd0 3fff40a4  
3ffffc40:  00000000 400042db 00000003 402157dd  
3ffffc50:  00000000 3ffffc90 00000004 00000000  
3ffffc60:  4010500c 40245f94 00000001 3fff0684  
3ffffc70:  00064020 00000000 00000001 40241740  
3ffffc80:  40258c90 fffffe7f 00000000 40207270  
3ffffc90:  3ffef558 3ffef558 3fff0684 4024b135  
3ffffca0:  4021f244 00000001 00000000 3fff0684  
3ffffcb0:  00064020 00000000 00000001 402201b8  
3ffffcc0:  00000000 3ffeff2c 00000019 402227f4  
3ffffcd0:  00000000 00000000 00000001 3ffefd20  
3ffffce0:  3ffefda0 3fff457d 00000001 4020d8c8  
3ffffcf0:  3fff00e9 3ffe8304 00000000 401004d8  
3ffffd00:  3fff27da 00000005 3ffefd20 40225e53  
3ffffd10:  00000000 000001bd 3ffeff2c 402210e0  
3ffffd20:  00000005 3ffe89cc 3ffefd20 3fff457d  
3ffffd30:  3fff27d4 00000000 3ffefd20 4020dafe  
3ffffd40:  00085580 3fff44dc 3ffecb85 4000050c  
3ffffd50:  40000001 000001bd 0000000f 00000000  
3ffffd60:  3fff457d 3fff395c 3ffffdc0 00000000  
3ffffd70:  3fff457d 3fff3954 00000001 4022266c  
3ffffd80:  00085580 0000c000 00000218 4020bf37  
3ffffd90:  40000001 3fff3944 00000001 40222b4a  
3ffffda0:  4010690e 0712f50f 401021f2 40214000  
3ffffdb0:  00085580 3ffeab58 3ffeab58 4000050c  
3ffffdc0:  3fff4094 00000030 4020bf14 40222644  
3ffffdd0:  40000001 00000000 000003e8 00000000  
3ffffde0:  3fff3954 3fff27d4 00000000 3fff1bf8  
3ffffdf0:  3fff197c 00000445 00000445 00000000  
3ffffe00:  3ffffe70 000001bd 3fff3a0c 402229e5  
3ffffe10:  000001bd 00000000 00085580 00000001  
3ffffe20:  3fff4522 00000218 3fff01f0 40215335  
3ffffe30:  000001bd 00000000 00085580 00000001  
3ffffe40:  3fff4522 00000218 3fff01f0 40213df5  
3ffffe50:  3fff1f34 00000000 402130d8 40222954  
3ffffe60:  3fff4094 3ffee064 4021308c 402229ac  
3ffffe70:  0000005b 00000030 0000001a ffffffff  
3ffffe80:  3fff0371 3fff3a0c 3fffdcc0 00000020  
3ffffe90:  3ffed920 3ffedb98 00000002 00000000  
3ffffea0:  3fff01f0 3fff44dc 00000000 40213e28  
3ffffeb0:  00000000 00000001 3fff1bf0 40215b1c  
3ffffec0:  3fff40a4 00000001 0000001c 3fffdab0  
3ffffed0:  3ffedb70 3fffdcc0 3ffea3f8 3fff1c10  
3ffffee0:  3fff40a4 3fff1bf4 3fff1bf0 40215b60  
3ffffef0:  3fff40a4 3fff1bf4 3fff1bf5 4022ce44  
3fffff00:  2701a8c0 00000000 3ffeb2ea 00000000  
3fffff10:  00000000 0000024e 3fffff80 3fff44fa  
3fffff20:  3fff1d68 3fff44dc 3fff1e50 40230d69  
3fffff30:  00000014 000003c6 000003c6 3fff1d68  
3fffff40:  3fffdc80 3fff230c 3fff3c9c 3fff236c  
3fffff50:  00000008 3fff1d68 3fff44dc 4022a3c9  
3fffff60:  3fffdc80 3fff230c 3fff3c9c 4010453c  
3fffff70:  4024797e 3fff230c 3fff3c9c 40247990  
3fffff80:  3fff44ec 3fff44dc 00000000 3fff0730  
3fffff90:  4024285b 00000000 3fff3c9c 402498ef  
3fffffa0:  40000f49 3fffdab0 3fffdab0 40000f49  
<<<stack<<<

 ets Jan  8 2013,rst cause:2, boot mode:(3,7)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v614f7c32
~ld

 ets Jan  8 2013,rst cause:4, boot mode:(3,7)

wdt reset
load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v614f7c32
~ld
⸮

Any new information about how to fix or workaround this?
I think this is crucial and should be investigated for version v2.1.0

luebbe · 2018-09-20T09:35:26Z

My workaround is to use normal OTA instead of Homie OTA...

sillyfrog · 2018-09-20T21:06:20Z

@timpur Do you have any thoughts as to where we could start looking at this issue? I'm not advanced in low level ESP development, but always happy to have a poke around - I just have no clue with this one where to start :/
Thanks!

timpur · 2018-09-25T22:22:58Z

Assuming your using latest SDK and latest 2.0 release

Maybe try the different lwip options from the SDK, I have a feeling the smallest lwip option from the SDK might work

Sorry I haven't gotten to this, been so busy with new job and all. But I'm slowly still working on homie, currently trying to get 2.1 out the door, might add this bug to that release.

CWempe · 2018-09-26T14:50:45Z

Maybe try the different lwip options from the SDK

I would like to try this. But I have no idea what that means and where to change this lwip option. 🙄

And would I need to change this for the existing firmware or the firmware that gets uploaded? (I can find that out myself)

Anyway.
Thanks and an I would like to see this fixed in 2.1. 😃

jmozmoz · 2018-09-26T22:13:38Z

Maybe try the different lwip options from the SDK
I would like to try this. But I have no idea what that means and where to change this lwip option. roll_eyes

If you use platformIO as build environment:
http://docs.platformio.org/en/latest/platforms/espressif8266.html#lwip-variant

If you use Arduino IDE:
Tools->lwip Variant -> 4 different options (v2 Low Memory, v2 High Bandwidth, ...)

CWempe · 2018-09-30T19:32:14Z

I have tested with Arduino IDE and lwip v1.4 now.

First uploaded my sketch via USB and then (only changed homie firmware version number) via homie-ota.

Did not work. 😞

Got this messages via mqtt:

2018-09-30 21:31:39 homie/DeckenlampeTEST/$implementation/ota/status 202
2018-09-30 21:31:39 homie/DeckenlampeTEST/$implementation/ota/status 500 INTERNAL_ERROR 11

sillyfrog · 2018-10-01T05:27:05Z

@CWempe Be sure to do a hard reset after flashing over serial to test OTA, there is an issue with OTA where it won't work reliably after getting flashed over serial (if I remember correctly - this is at least what I observe). Be sure to select v2 Lower Memory.

For me, following the directions at http://docs.platformio.org/en/latest/platforms/espressif8266.html#lwip-variant , and using PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY exactly as per the example, things work great!

Thank you everyone! This appears to be sorted for me at least.

CWempe · 2018-10-01T21:31:27Z

Still no luck. 😞

I did a hard reset after every serial flash, now.
I tried LwIP v1.4 and 2.0 lower memory.

CWempe · 2018-10-01T21:32:19Z

btw: This could be a duplicate of #261

CWempe · 2018-10-05T23:08:25Z

I started using PlatformIO now.
Still no success with PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY. 😞
But I am not sure how to check if this option is actually be used while compiling. 🤔

Hopefully this will be fixed with this PR. (LwIP 2.1.0)
esp8266/Arduino#4980 (comment)

sillyfrog · 2018-10-05T23:13:08Z

@CWempe After compiling, are you flashing via serial, then trying an OTA update?

I'm not sure on how to check in the compiled code, but one thing I did is change the PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY to something broken (eg: PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORYXXX), and try to compile, and ensured it complained that it was invalid. For me, once flashing over serial, and doing a hard reboot, OTA was working, but my sketch is not very big.

JorinL · 2018-10-22T09:17:20Z

@sillyfrog can you please try to execute this cmd (from windows/linux) to the ip of your Wemos Device:
(it is nothing else than the invitation to OTA for the device)

echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/IP-OF-YOUR-WEMOS/8266

Please take a look into you mqtt in the same time and check if it resets everytime you send the command from above.

Please tell me if it resets everytime you send it.

If it does, then please check your config.json if you named your device in a poattern like "sonoff-dual-01" and if so remove the zero.

I can say to 100% that the name "sonoff-dual-01" causes reset in the moment that the device gets the OTA invitation. I will open a Bug for that - I'm currently testing if it is Homie oder Arduino-OTA where I have to do that.

sillyfrog · 2018-10-23T02:35:24Z

I have done that now, both with the PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY option and without, and there is no change.

I have also done it against port 1883 (I don't know anything about this UDP process, so giving things a test). But either way, nothing happened (with or without the PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY option, and either port).

This was on a D1 Mini Pro v1.1.0, using the TemperatureSensor example.

Cheers.

JorinL · 2018-10-23T12:16:11Z

No reset did happen with echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/IP-OF-YOUR-WEMOS/8266 ?

Mhh and what is your configure device-id?

Sounds like another problem than I'm thinking of.

sillyfrog · 2018-10-24T00:05:22Z

For this test, it was named testdev.

JorinL · 2018-10-24T08:09:29Z

And your problem still persists? If using OTA the device crashes and restarts (I try to avoid reset - cause this is a Homie function which also deletes the config).

I use the Wemos D1 mini and cought a problem with ota names crashing the device when trying to use ota.

sillyfrog · 2018-10-24T21:11:58Z

I have no issues if I set the PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY compile time option (and with that in mind, I might put in a PR so that's included with the default platformio.ini)

timpur · 2018-10-25T02:22:19Z

@sillyfrog I thought you said PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY didn't work ?

sillyfrog · 2018-10-25T02:33:50Z

@timpur Using the PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY did work for me see this comment.

But sending the raw UDP packet, did not cause my device to crash, things kept working, with or without the PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY build flag (I'm not sure my device was even listening on this UDP port).

Thanks again for your help on this!

timpur · 2018-10-25T07:55:47Z

Done. So can we close this ? @sillyfrog

CWempe · 2018-10-25T08:35:20Z

I am still having this problem.
But I am not home at the moment.
I hope to try your tests next week.

sillyfrog · 2018-10-25T10:20:23Z

@timpur For me, yes, but not sure about others (as per @CWempe)

Cheers.

CWempe · 2018-10-28T21:34:58Z

I just issued the command to see if my devices would reset.

echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/192.168.1.12/8266
echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/192.168.1.15/8266
echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/192.168.1.42/8266

No rest.

My devices are named:

DeckenlampeAZ
DeckenlampeWZ
FoodTracker

targetblank · 2019-11-05T19:54:10Z

I can confirm and reproduce this issue on a Wemos D1 mini both with Homie OTA and ArduinoOTA. All test with build flag PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY
It is related to the device_id as @JorinL suggested.

device_id: homie-alu-led-2

echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/192.168.2.12/8266 resets the device
ArduinoOTA not working (Exception (3) ...)
Homie OTA not working (Exception (3) ...)

device_id: led2

echo "0 48003 412864 50a61345aed86f6dabce66fca2de94a2" >/dev/udp/192.168.2.12/8266 does not reset the device
ArduinoOTA working
Homie OTA working

So for me it is certain device_ids that break the process. As possible workaround maybe you can use lowercase ids without any special chars. And I think further investigation in ArduinoOTA library is necessary in that case.

luebbe · 2019-11-06T12:59:08Z

Very interesting. Is it the combination of dash and number in the device id?

JorinL · 2019-11-06T19:19:31Z

I did not found out, what exactly the cause was - I changed my devicenaming after I found out.

From "sonoff-s20-06" to "sonoff-s20-6" the "0" seems not to be the thin if we look at @targetblanks post. So the "-" should do it in a specific way - because at my side they work :-D

But I'll test ist again. Had some OTA problems - if leaving out the dash fixes it.. :-D

luebbe · 2019-11-08T08:45:47Z

Perhaps name-<number> is the lethal part. Combinations with name-<string> work and name<number> works too.
A wild guess is that <-01> evaluates to a negative number somewhere and causes an integer overflow.

CWempe · 2019-11-08T09:07:29Z

I am not sure, because it was a while ago.
But I think my device names had no number or dash in them.

luebbe · 2019-11-08T14:40:15Z

Sorry, I didn't notice that the brackets got messed up. Fixed the comment

CWempe · 2019-11-16T20:50:10Z

I just tested again without success. 😞

Just to be clear: I am using the develop branch (v2.0.1).

This is my device:

homie/sideboard/$homie 2.0.1
homie/sideboard/$name sideboard
homie/sideboard/$mac 5C:CF:7F:D9:7E:FC
homie/sideboard/$localip 192.168.1.46
homie/sideboard/$nodes temperature,light01,light02,light03,light04,led,power
homie/sideboard/$stats/interval 0
homie/sideboard/$fw/name test
homie/sideboard/$fw/version 9.0.1
homie/sideboard/$fw/checksum 59488442281c84be1d11deb7c7b2cdfa
homie/sideboard/$implementation esp8266
homie/sideboard/$implementation/config {"wifi":{"ssid":"CWempe"},"mqtt":{"host":"192.168.1.140","port":1883,"auth":true},"name":"sideboard","ota":{"enabled":true},"device_id":"sideboard"}
homie/sideboard/$implementation/version 2.0.0
homie/sideboard/$implementation/ota/enabled true

I tried updating the same sketch with an incremented version number (9.0.2) via homie-ota.

But the device crashes:

Receiving OTA payload
↕ OTA started
Triggering OTA_STARTED event...
Firmware is base64-encoded
Receiving OTA firmware (338/551730)...
Receiving OTA firmware (740/551730)...

Abort called

>>>stack>>>

ctx: sys
sp: 3fffeaa0 end: 3fffffb0 offset: 01b0
3fffec50:  3fff057c 000004b0 3ffefa68 40238f0c
3fffec60:  00000218 3ffefa68 3ffefb14 4020704f
3fffec70:  00000218 40238f3a 0000004e 40238efa

[..]

3fffff70:  00000000 00000000 ff223f14 00000000  
3fffff80:  00000000 ff229878 00000000 40201d39
3fffff90:  00000000 00000000 00000001 3fff04e0  
3fffffa0:  3fffdad0 00000000 3fff04b0 40210050  
<<<stack<<<

last failed alloc call: 40238F3A(735641)

 ets Jan  8 2013,rst cause:2, boot mode:(3,6)

load 0x4010f000, len 1384, room 16 
tail 8
chksum 0x2d
csum 0x2d
v8b899c12
~ld

I compiled the sketch with PIO.

[env:d1_mini_pro]
board = d1_mini_pro
platform = espressif8266
framework = arduino
monitor_speed = 115200
build_flags = -D PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY

lib_deps =
     https://github.com/homieiot/homie-esp8266#develop
     elapsedMillis
     NeoPixelBus
     DallasTemperature

Same without build_flags = -D PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY.

And I just tried the same with homie develop-v3. Same error.
And tried the example sketch (https://github.com/homieiot/homie-esp8266/blob/develop-v3/examples/LightOnOff/LightOnOff.ino), too.

I reseted my device and left the device-id to default. No success.

So in conclusion:

I could not fix the issue with device name without special characters.
I could not fix the issue with device name "test".
I could not fix the issue with default device id.
I could not fix the issue with the latest homie version
I could not fix the issue with the example sketch
I could not fix the issue by reseting config.
I could not fix the issue by using the official ota_updater.py (develop-v3) instead of homie-ota

JorinL · 2019-11-19T16:43:00Z

Hey there.

For me it looks like OTA-Update from older Versions or OTA in older Versions has a Problem.

With a fresh via serial flashed SonOff S20 (esp8266) using eps8266 in version 2.61
And using homie-esp8266-develop-v3 it works for me again.

Updating to 2.61 and dev-v3 makes exactly that crash - i think i used something like 2.42 and homie 2.0.0 before - and from that it seems only updating with serial connection works - but after that OTA is fine.

CWempe · 2019-11-20T11:28:12Z

Just to be clear.
I tried updating from a freshly flashed develeop-v3 (Serial) to develeop-v3 (OTA).
But it did not work.

I used PlatformIO.
I am not sure wich eps8266 version PlatformIO is/was using. 🤔
How can I check that?

I think it is 2.2.3.

But that sounds rather old.

CWempe · 2019-11-20T15:35:19Z

I think these issues confirm that the "official" PlatformIO package is outdated.
platformio/platform-espressif8266#181
platformio/platform-espressif8266#167
platformio/platform-espressif8266#85

This might be a workaround.
https://github.com/arendst/Tasmota/blob/development/platformio.ini#L164-L171
I will try this later.

JorinL · 2019-11-20T15:47:21Z

Hehe I use Arduino IDE - where i can plug version together as I please :-D

(has bad and has good sides) :-D
I verified my idea from before - flashing serial on the old with new versions - brings ota back (using arduino ota).

CWempe · 2019-11-20T22:11:14Z

I did some testing now. But no succcess. 😞

I tried OTA from essif8266 2.6.0 to essif8266 2.6.0 (the same version).
I tried OTA from essif8266 2.6.0 to essif8266 2.6.1.
I tried OTA from essif8266 2.6.1 to essif8266 2.6.1 (the same version).

This is my platformio.ini:

[env:d1_mini_pro]
; *** Esp8266 core for Arduino version 2.6.0 (for Windows, most Linux variants and Mac)
; *** custom setup until the core 2.6.0 version is official released from PlatformIO crew
platform                  = https://github.com/Jason2866/platform-espressif8266.git#core_2_6_1
board                     = d1_mini_pro
build_flags               = -D NDEBUG
                            -D PIO_FRAMEWORK_ARDUINO_LWIP2_LOW_MEMORY
                            -mtarget-align
                            -O2
                            -DBEARSSL_SSL_BASIC
                            ; -Teagle.flash.1m.ld
                         ; enable one option set -> No exception recommended
                         ; No exception code in firmware
                            -fno-exceptions
                            -lstdc++

Maybe I will try without PlatformIO tomorrow.

luebbe · 2019-11-21T08:18:36Z

I'm using VSCode with the PIO plugin and I don't remember switching to a different version of the expressif core.
I had problems with Arduino OTA a few weeks ago. The reason for this was that since some update of windows or VSCode, VSCode has to be allowed through the windows firewall explicitly for OTA to work.
The error message in PIO was along the lines of "... authentication failed ..."

CWempe · 2019-11-21T08:31:08Z

I am using homie-ota for OTA.
I am using VSCode and PIO too, but I did not try the ota-feature in VSCode, yet.

@luebbe
So it is working for you?
What espressif8266 version are you using? (you can see that in the build output)
What hardware are you using? "D1 mini", "D1 mini pro", ...?
What are your build_flags?

luebbe · 2019-11-22T20:03:13Z

It's a D1 mini, but ota also works with an off the shelf 12E or Sonoff devices. The output is:

CONFIGURATION: https://docs.platformio.org/page/boards/espressif8266/d1_mini.html
PLATFORM: Espressif 8266 2.2.3 > WeMos D1 R2 and mini
HARDWARE: ESP8266 80MHz, 80KB RAM, 4MB Flash
PACKAGES: toolchain-xtensa 2.40802.190218 (4.8.2), tool-esptool 1.413.0 (4.13), tool-mkspiffs 1.200.0 (2.0), tool-esptoolpy 1.20600.0 (2.6.0), framework-arduinoespressif8266 2.20502.0 (2.5.2)
LDF: Library Dependency Finder -> http://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft

So it's a rather old version of the expressif SDK

luebbe · 2019-11-26T10:34:20Z

Perhaps related to #526?

* Lets try to fix travis * fix paths * missed one path * some more fixes * try with copy? * no log? * Now fix the second sketch

stritti · 2020-12-28T11:15:49Z

Is this issue solved @sillyfrog? Could it be closed by now?

sillyfrog · 2020-12-28T11:50:27Z

@stritti Yes, I've been using this (well the current release at least) for some time and it's still working for me. Thanks!

jmozmoz mentioned this issue Sep 30, 2018

OTA does not work if TLS/SSL is enable #532

Open

gorec2005 pushed a commit to gorec2005/homie-esp8266 that referenced this issue Jun 28, 2020

Fix travis (homieiot#530)

f5b51fa

* Lets try to fix travis * fix paths * missed one path * some more fixes * try with copy? * no log? * Now fix the second sketch

sillyfrog closed this as completed Dec 28, 2020

OTA Crashing WeMos device #530

OTA Crashing WeMos device #530

Comments

sillyfrog commented Jun 24, 2018 • edited Loading

timpur commented Jun 24, 2018

sillyfrog commented Jun 25, 2018

CWempe commented Sep 19, 2018

luebbe commented Sep 20, 2018

sillyfrog commented Sep 20, 2018

timpur commented Sep 25, 2018 • edited Loading

CWempe commented Sep 26, 2018

jmozmoz commented Sep 26, 2018 • edited Loading

CWempe commented Sep 30, 2018

sillyfrog commented Oct 1, 2018

CWempe commented Oct 1, 2018

CWempe commented Oct 1, 2018

CWempe commented Oct 5, 2018

sillyfrog commented Oct 5, 2018

JorinL commented Oct 22, 2018 • edited Loading

sillyfrog commented Oct 23, 2018

JorinL commented Oct 23, 2018 • edited Loading

sillyfrog commented Oct 24, 2018

JorinL commented Oct 24, 2018 • edited Loading

sillyfrog commented Oct 24, 2018

timpur commented Oct 25, 2018

sillyfrog commented Oct 25, 2018

timpur commented Oct 25, 2018

CWempe commented Oct 25, 2018 • edited Loading

sillyfrog commented Oct 25, 2018

CWempe commented Oct 28, 2018

targetblank commented Nov 5, 2019 • edited Loading

luebbe commented Nov 6, 2019

JorinL commented Nov 6, 2019 • edited Loading

luebbe commented Nov 8, 2019 • edited Loading

CWempe commented Nov 8, 2019

luebbe commented Nov 8, 2019

CWempe commented Nov 16, 2019

JorinL commented Nov 19, 2019

CWempe commented Nov 20, 2019

CWempe commented Nov 20, 2019

JorinL commented Nov 20, 2019 • edited Loading

CWempe commented Nov 20, 2019

luebbe commented Nov 21, 2019 • edited Loading

CWempe commented Nov 21, 2019

luebbe commented Nov 22, 2019 • edited Loading

luebbe commented Nov 26, 2019

stritti commented Dec 28, 2020

sillyfrog commented Dec 28, 2020

sillyfrog commented Jun 24, 2018 •

edited

Loading

timpur commented Sep 25, 2018 •

edited

Loading

jmozmoz commented Sep 26, 2018 •

edited

Loading

JorinL commented Oct 22, 2018 •

edited

Loading

JorinL commented Oct 23, 2018 •

edited

Loading

JorinL commented Oct 24, 2018 •

edited

Loading

CWempe commented Oct 25, 2018 •

edited

Loading

targetblank commented Nov 5, 2019 •

edited

Loading

JorinL commented Nov 6, 2019 •

edited

Loading

luebbe commented Nov 8, 2019 •

edited

Loading

JorinL commented Nov 20, 2019 •

edited

Loading

luebbe commented Nov 21, 2019 •

edited

Loading

luebbe commented Nov 22, 2019 •

edited

Loading