[WIP][RFT] generic: platform/mikrotik: add wlan lz77 decompress

[john-tho]

usual disclaimer: expect your device to halt and catch fire…

very rough attempt to decode lz77; rough, excessively verbose, and entirely unoptimised.

A number of new (or newly factory flashed) Mikrotik devices are using LZ77 magic for wlan tag hard_config data.
New devices include the Chateau LTE12 1, and ax devices 2 Newly factory flashed devices may include the hap ac3 3

This can be seen in decoded OEM supout 4 dmesg:
"radio data lz77 decompressed from"…

Investigating an arm RouterOS flash.ko module, and supplied example hard_config dumps, the format was determined via 5.

[john-tho]

Suggested change

	#ifdef CONFIG_CPU_LITTLE_ENDIAN
	#if defined (CONFIG_CPU_LITTLE_ENDIAN) || defined(CONFIG_ARCH_IPQ40XX)

ipq40xx does not set CONFIG_CPU_LITTLE_ENDIAN

[robimarko]

I think it does in the final kernel config

[dchard]

The latest push is tested on Mikrotik Chateau-12LTE (ipq4019). Both radios are working, driver does load the firmware with BDF/CAL properly, WLAN MACs are also correct. Test is conducted on Routerboot_v7 with the required additional patches. Dmesg is completely fine, no errors or weird messages.

firmware ver 10.4b-ct-4019-fW-13-5ae337bb1 api 5

If you want you can use my "tested by" tag:
Csaba Sipos <metro4_at_freemail.hu>

[john-tho]

Hopefully no functional changes with push, only fixing tidy messages, and fix some checkpatch and sparse warnings

[dchard]

Hopefully no functional changes with push, only fixing tidy messages, and fix some checkpatch and sparse warnings

I can recheck it early next week.

[dchard]

I tested it on ipq4019 (chateau 12) and it still works fine.

[dchard]

@john-tho found a device out of the many that fails at LZ77 decompress:

[    0.960517] [rb_hardconfig][lz77] input overrun
[    0.960565] [rb_hardconfig] LZ77: LZ77 decompress fail
[    0.964427] [rb_hardconfig][lz77] input overrun
[    0.969139] [rb_hardconfig] LZ77: LZ77 decompress fail
[    0.974058] [rb_hardconfig][lz77] input overrun
[    0.978735] [rb_hardconfig] LZ77: LZ77 decompress fail
[    0.983165] MikroTik RouterBOARD hardware configuration sysfs driver v0.08
[    0.992248] MikroTik RouterBOARD software configuration sysfs driver v0.05

And this is the MTD2 (hard config) dump of the same device:

[deleted]

[john-tho]

@john-tho found a device out of the many that fails at LZ77 decompress:

[    0.960517] [rb_hardconfig][lz77] input overrun
[    0.960565] [rb_hardconfig] LZ77: LZ77 decompress fail
[    0.964427] [rb_hardconfig][lz77] input overrun
[    0.969139] [rb_hardconfig] LZ77: LZ77 decompress fail
[    0.974058] [rb_hardconfig][lz77] input overrun
[    0.978735] [rb_hardconfig] LZ77: LZ77 decompress fail
[    0.983165] MikroTik RouterBOARD hardware configuration sysfs driver v0.08
[    0.992248] MikroTik RouterBOARD software configuration sysfs driver v0.05

And this is the MTD2 (hard config) dump of the same device:

LZ77_decomp_failed_mtd2_hard_config.zip

Okay, thanks, I pulled a copy.

In my standalone decompressor, looks like the final group (non-match) bits have some set bits.

inbit:0x55d1->outbyte:0x2e8a bits opcode:0xc match, offset: 0x1, len: 0x6 (5 partial memcpy)
inbit:0x55dd->outbyte:0x2e90 bits opcode:0xc non-match, len: 0xc
 correct len
(input_bit 0x55e9 + opcode->length 0xc*8) = 0x5649 > in_len 0xac0 (bits 0x5600)
 correct end
input bit 0x55e9, input_len * 8: 0x5600
input byte -1 0x0, 0xe
input bit 0x55dd
opcode bits
 1 1 0 0 0 0 0 0 0 0 0 0
non-match data bits
 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
input overrun
finished, rc=-1
output first bytes:
44 52 45 00 01 82 28 19

You could try removing the && (!lz77_mikrotik_wlan_get_bit(in, input_bit)) from the test end marker section.
It might just work, and my null bit(s) follows last group opcode assumption (based on all prior examples) was wrong.

We should look to see how the Mikrotik binary handles this last group & payload.

[john-tho]

In my standalone decompressor, looks like the final group (non-match) bits have some set bits.

inbit:0x55d1->outbyte:0x2e8a bits opcode:0xc match, offset: 0x1, len: 0x6 (5 partial memcpy)
inbit:0x55dd->outbyte:0x2e90 bits opcode:0xc non-match, len: 0xc
 correct len
(input_bit 0x55e9 + opcode->length 0xc*8) = 0x5649 > in_len 0xac0 (bits 0x5600)
 correct end
input bit 0x55e9, input_len * 8: 0x5600
input byte -1 0x0, 0xe
input bit 0x55dd
opcode bits
 1 1 0 0 0 0 0 0 0 0 0 0
non-match data bits
 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
input overrun
finished, rc=-1
output first bytes:
44 52 45 00 01 82 28 19

You could try removing the && (!lz77_mikrotik_wlan_get_bit(in, input_bit)) from the test end marker section. It might just work, and my null bit(s) follows last group opcode assumption (based on all prior examples) was wrong.

We should look to see how the Mikrotik binary handles this last group & payload.

Emulated binary decompressed payload was 0x2e90 bytes long, so I should be able to cut the extra test for bits following the last group opcode: && (!lz77_mikrotik_wlan_get_bit(in, input_bit))

[dchard]

@john-tho just to be sure: this does not looks like a potential flash/data corruption to you? I used your patch on about 80 devices from multiple batches of Chateau LTE12 and there was a single one so far which produced this fault.

[john-tho]

@john-tho just to be sure: this does not looks like a potential flash/data corruption to you?

Does not look like corrupted flash. If it was, this lz77 decompress would fail very differently. Just extra set bits after the last (non) match group.

[dchard]

@john-tho just to be sure: this does not looks like a potential flash/data corruption to you?

Does not look like corrupted flash. If it was, this lz77 decompress would fail very differently. Just extra set bits after the last (non) match group.

Tested your latest modifications on the affected device, and now it works without any errors.

[aanon4]

How little endian is this decompression code? I have two ath79 based device (921GS-5HPacD-19s - one of their newer sectors) and one boots fine while the other doesn't load the firmware. I was thinking this compression scheme might be the problem there also? However this device is big endian so I wondered how portable it was?
Thanks

[Edit]
Looking though the hard_config data, it is definitely LZ77 compressed.

[Edit 2]
I tried it "as is" and it works great! Thanks so much. You might want to consider widening the scope of where this is included.

[john-tho]

How little endian is this decompression code? I have two ath79 based device (921GS-5HPacD-19s - one of their newer sectors) and one boots fine while the other doesn't load the firmware. I was thinking this compression scheme might be the problem there also? However this device is big endian so I wondered how portable it was? Thanks

[Edit] Looking though the hard_config data, it is definitely LZ77 compressed.

[Edit 2] I tried it "as is" and it works great! Thanks so much. You might want to consider widening the scope of where this is included.

Ha! Colour me surprised. I was not expecting to see this used on MIPS. Many thanks for the new data and testing Tim. I guess, because the compressed data is read one bit at a time, that should have been the only place where endian-ness mattered. If you could, I would appreciate you detailing the backup RouterBOOT version so we might be able to piece together some idea of when this started: cat /sys/firmware/mikrotik/hard_config/booter_version

Okay, I will adjust the LE|IPQ40XX build conditional next time I touch this.

Cheers.

[aanon4]

The output from that cat command:
6.48.6

[dchard]

@john-tho just checked the LZ77 and routeros-v7 patches on the new 6.1.55 kernel for IPQ4019 and it seems to work correctly.

[MichaelUray]

The patch works for me with a MikroTik hAP ac3.
What is actually required to get that merged into the master?

[john-tho]

Hi @aanon4,

I have two ath79 based device (921GS-5HPacD-19s - one of their newer sectors) and one boots fine while the other doesn't load the firmware.
[Edit] Looking though the hard_config data, it is definitely LZ77 compressed.

I am gradually getting this together for upstream OpenWrt (openwrt#15774), and have made a few small changes to the code. I would be interested to see if you could try out this updated code, and/or email me a copy of hard_config and soft_config mtd partitions for that lz77 ath79 device?

The output from that cat command:
6.48.6

There are a number of Mikrotik ath79 devices with ath10k pcie cards, and I have not noticed any other OpenWrt forum or issue reports of no 5GHz Wi-Fi on them (but for some time they have had more modern replacements). My guess now is that this may occur (for ath79) if calibration data is updated (guessing in RouterOS update) for US devices to enable U-NII 2 frequency ranges, or recent factory flashes.

[aanon4]

Hi @aanon4,

I have two ath79 based device (921GS-5HPacD-19s - one of their newer sectors) and one boots fine while the other doesn't load the firmware.
[Edit] Looking though the hard_config data, it is definitely LZ77 compressed.

I am gradually getting this together for upstream OpenWrt (openwrt#15774), and have made a few small changes to the code. I would be interested to see if you could try out this updated code, and/or email me a copy of hard_config and soft_config mtd partitions for that lz77 ath79 device?

The output from that cat command:
6.48.6

There are a number of Mikrotik ath79 devices with ath10k pcie cards, and I have not noticed any other OpenWrt forum or issue reports of no 5GHz Wi-Fi on them (but for some time they have had more modern replacements). My guess now is that this may occur (for ath79) if calibration data is updated (guessing in RouterOS update) for US devices to enable U-NII 2 frequency ranges, or recent factory flashes.

@john-tho Unfortunately this is tricky. I work on the AREDN project which is built on top of OpenWRT. As such the radio in question is currently at the top of a tower at the top of a mountain. I can't really run test code on it because ... well ... getting to it if fails is a massive pain.

[john-tho]

@john-tho Unfortunately this is tricky. I work on the AREDN project which is built on top of OpenWRT. As such the radio in question is currently at the top of a tower at the top of a mountain. I can't really run test code on it because ... well ... getting to it if fails is a massive pain.

Haha, yes, no worries, kind of expected this. I too have not-so-easily-accessible devices. Lab test devices are boring, field tests (or deploys) are much more interesting.
If someone could possible remote into the device, and pull a dump of the mtd hard_config partition, that would allow me to run some sanity checks on my code at least. soft_config could just tell us if RouterBoot primary had been updated as well.
Also, please let me know if you heard of any other ath79 devices that needed this (used LZ77 calibration data)?
Cheers.