Direct Stepping feature

[colinrgodsey]

Description

This is a revival of the Marlin 1 PR for "stepper chunks" #7047

Direct Stepping allows a host device to issue direct stepper movements that are defined in a binary format that is written ahead of time by the host device to a page in the device RAM.

The mechanism provided for loading the binary data onto the device is the "page manager". The page manager provided here uses a parallel protocol over the USB serial connection to write pages to the device RAM, and is decoupled from the normal serial g-code pipeline. This allows the host device to preload pages as fast as possible without waiting on the g-code pipeline.

Once a page is written by the device, it can be triggered using the G6 g-code which references the page index that should be used for that move. Depending on the page format, direction arguments may need to be provided in the G6 code.

Page Formats

Each format has different tradeoffs and performance characteristics. This PR includes 3 different page formats that should cover a general range of use cases, as well as an abstraction for creating different formats. The naming scheme for the formats follow this general pattern:

SP_<axes>_<compression ratio><'D' if directional>_<number of segments>

The three provided formats:

• SP_4x4D_128 - 4 axes, 4x compression, direction is binary encoded, 128 segments, 1024 steps and 256 bytes per page.
• SP_4x2_256 - 4 axes, 2x compression, no binary direction, 256 segments, 1024 steps and 256 bytes per page.
• SP_4x1_512 - 4 axes, no compression, no binary direction, 512 segments, 512 steps and 256 bytes per page.

The segment-based compression still provides total geometric accuracy, but breaks the motion into tiny line segments for the sake of page size. The compressed segment value is then used in a lookup table to produce the actual steps the device will take.

SP_4x4D_128 is a good format for movements which may be very quick and change direction often (multiple times per 1024 steps). SP_4x2_256 is a good general purpose format that seems optimal for 3d printing. SP_4x1_512 may not be able to reach the same step rate as the other two, but provides total motion accuracy.

More detailed description of the page formats and the serial page manager protocol will be provided soon on the RepRap wiki: https://reprap.org/wiki/Direct_Stepping

G6 Code

The g-code used to trigger the direct stepping motion references the index of the written page, and varies slightly if the format is directional:

G6 
I - Page index
S - Number of steps to take. Defaults to max steps.
R - Step rate per second. Last value is cached for future invocations. 
X, Y, Z, E - Direction: 1 for positive, 0 for negative. Last 
value is cached for future invocations. Not used for 
directional formats. 

Benefits

Ultimately this feature is about offloading the kinematics, physics and planning to an external device. This allows Marlin to run more in a deterministic and constant-time state by pushing the bulk of the complexity and mathematics on to a more powerful general purpose computing device.

Testing

Testing can be done using step-daemon.

WARNING: Position sync in step-daemon is not entirely finished and movements made with the control panel may be lost. At this moment, homing is required to sync step daemon and the device. Always home before issuing movement commands.

[thinkyhead]

I managed to rebase and squash this using the good old merge-and-reset trick. No problemo. To update your working copy:

git checkout direct_stepping
git fetch origin
git reset --hard origin/direct_stepping

[colinrgodsey]

I managed to rebase and squash this using the good old merge-and-reset trick. No problemo. To update your working copy:

Awesome, thanks!

[colinrgodsey]

Example of host transmission for SP_4x2_256:

https://gist.github.com/colinrgodsey/f35e4f33eb3cdd80cfb985362703db03

[colinrgodsey]

finally got a real print going: https://twitter.com/colinrgodsey/status/1259264651235225601

Had some issues with stepd to clear up, but it should be good to use for testing now, with a few caveats:

1. You can not use the panel between homing and printing. It sucks, but right now stepd only syncs after homing.

2. Stepd does not have any max axis velocity settings. So make sure any pre-print movements (like parking the Z axis high), should be accompanied with a slow feed rate movement back to near Z0 before printing (at least with my lead screw, the velocity goes too high and it just locks up).

Also with my current testing, I'm using the SP_4x2_256 format at 61440 steps/s. Works just fine on the mega2560.

[thinkyhead]

Once this is merged, will you put together an "easy" install package for OctoPrint setups? It would be great to see some of those older simpler machines get some new life.

[colinrgodsey]

@thinkyhead yeah absolutely. Once we get everything squared away on the marlin side, ill make a dpkg script or something for stepd to get it all more convenient

[thinkyhead]

Ok, so to get this ready to deploy I've applied some cleanup to match our fun standards, and…

• Put the segment_table into PROGMEM.
• I noticed that the SerialPageManager data and methods could all be made static since there will only be one instance of this object. Doing that eliminated the hidden "this" parameter that was being added to all its methods, making code slightly smaller and faster.

[colinrgodsey]

@thinkyhead was watching the changes come in, looks good, and thanks for cleaning that up!

Didn't think about the hidden "this", makes lots of sense. I'll have to see if i can get the steprate even higher now.

[colinrgodsey]

Looks like there's some syntax issues in the new static serialpagemanager. got em fixed on my side. ill do some testing and commit em up 👍

[colinrgodsey]

Headless test was fine. Started a real test print (60kHz), seems fine so far 👍

[colinrgodsey]

Test print was fine until I got a nasty filament jam. cries in PLA

[thinkyhead]

Ouch. I just read a painful amount of information about typename.

[thinkyhead]

Once this is published, prepare for a lot of users asking
"Does this mean I can use Klipper and Marlin together?"
Maybe it's possible, but I bet it's not easy.

[colinrgodsey]

Yea, typename is a pretty brutal addition to C++... i guess it makes sense, but it sucks having to use it basically everywhere.

And.... yea, getting the marlin changes done i think will probably be the "easy" part of this whole venture. Usability and support will be another journey.

[thinkyhead]

This makes for informative reading, if you haven't already…
https://github.com/KevinOConnor/klipper/blob/master/docs/Protocol.md

The Klipper universe has me installing simulavr, which I should have installed long before now anyway…

[colinrgodsey]

Managed to get everything running up to 69,632 Hz. The harmonic noise starts to get pretty interesting at that speed... (still not loud, but Id like to aim for a white noise characteristic). Might play around with introducing some sort of entropy there for table-lookup offsets.

[thinkyhead]

I look forward to giving it a proper run once I have a free day to get everything set up.

[RaabenF]

amazing stuff. really. so you worked that out 3 years ago?

think i found the G6 in the recent bugfix, right? So this is just installing an octo addon and it will basically work?

im not really into that raspberry/linux stuff, my friend bought me a pie zero... thats shitty from the beginning. and i always wondered, how that stepping workload could be moved away from the controller

so sad that it took so long to publish that feature. is nobody interested? lucky for me, that im just printing for one year now. but that work seems to be a mature improvement right?

[howard0su]

As the recent Marlin board are more strong, like ESP32, STM32F4, etc. they have FPU in CPU. Are they capable to do the calculation on board instead of host machine?

[EvilGremlin]

Always had been. Only AVR and STM32G0 can struggle a bit when you go above 100mm/s, depending on other circumstances.