New LED hardware driver: WLED Pixel Bus

[DedeHai]

This is a major update adding WLED Pixel Bus written from scratch to replace NeoPixelBus and it packs a lot of useful features. From a user perspective the main improvements are less memory use and dynamic adjustment of LED timing to eliminate flickering by fine-tuning the timing from -30% to + 30% in 10% steps (dropdown menu).

The new hardware driver structure allows for fully dynamic updates of LED outputs and LED timings. It also adds glitch free parallel SPI output support on the C3 as well as parallel bit-banging on all platforms including ESP8266.
All parallel outputs (except bit-bang) use ping-pong DMA buffers instead of fully pre-filled buffers - memory usage is optimized and DMA buffers are independent of number of LEDs. Also the calculation from LED colors to DMA buffers is highly speed optimized, making the driver blazing fast - I saw 30% FPS improvements in some cases.
The fully dynamic nature of the driver allows for a "Custom digital bus" in the LED settings to support virtually any LED bus type out there - specify timing, number of color channels, invert any color channel or set any color channel combination. The driver also supports inverting the output signal on any pin (ESP32 only).

The LED config UI was updated to fully support the new driver options.

A lot more testing is required for all different kind of LEDs on all platforms. I am sure there is a ton of bugs and kinks to iron out.

I would like to take this opportunity to thank @Makuna for his outstanding work on NeoPixelBus which served me well as a reference on hardware configurations and special LED types.

• saves over 10k of heap on ESP32 and about 3k on ESP8266
• increases available LED outputs on ESP32 C3 from 2 to 14 (2x RMT, 4x parallel SPI, 8x bit-bang)
• increases available LED outputs on ESP32 S2 with 8x bit-bang if I2S is not available
• increases available LED outputs on ESP8266 3 to 6 (1x UART, 1xI2S DMA, 4x bit-bang)
• allows over 1000 LEDs again on ESP8266 thanks to reduced RAM footprint, it does not use a global buffer but writes to LED buffers directly which is slower but uses less RAM.
• adds support for TM1815
• improved low-brightness color accuracy to TM1814, TM1815 and APA102 by utilizing LED current settings
• improved color resolution on 16bit LED types (UCS8903, UCS8904, SM16825)
• if needed I2S (ESP32 and ESP32 S2) and LCD (ESP32 S3) driver can output on 16 parallel pins changing a #define
• automatic calculation and optimized use of multiple RMT memory blocks, reducing ISR frequency when not using all RMT outputs and thus increases FPS

ESP32 flash and RAM usage comparison (bit bang disabled)

using NeoPixelBus
RAM: [== ] 24.9% (used 81576 bytes from 327680 bytes)
Flash: [======== ] 83.1% (used 1306493 bytes from 1572864 bytes)
Free Heap: (6 outputs, 256 each): 79.8k

using WLEDpixelBus
RAM: [== ] 24.9% (used 81680 bytes from 327680 bytes)
Flash: [======== ] 83.0% (used 1306053 bytes from 1572864 bytes)
Free Heap: (6 outputs, 256 each): 92.8k

Summary by CodeRabbit

• New Features
  • Added per-bus Bus Speed Factor (SF/bsf) settings.
  • Added Custom Bus support with channel mapping, inversion, custom timing, and per-channel colors.
  • Expanded supported output driver/type options (bit-bang, parallel SPI, RMT, I2S, LCD where available) and added new LED type options (e.g., TM1815 / custom bus).
• Bug Fixes
  • Improved ESP8266 direct-to-bus pixel handling and refined gamma/brightness behavior across realtime and particle rendering.
• Chores
  • Updated capability limits and the LED settings UI + config import/export for the new bus features.

[willmmiles]

OK! I should be able to get some time to do a detailed review on the weekend.

[willmmiles]

I'm finding it challenging to review this properly - I just keep exploding in a hundred nitpicks. It's hard to know where to start and I feel bad pushing suggestions in dribs and drabs.

Bigger things I've noticed so far:

• WLEDpixelBus.h has duplicate copies of code from colors.h. This component already isn't isolated from WLED internals (it's importing const.h for WLED's type mapping). If we want to maintain the conceit that this is an independent library, pull out colors.h to its own "library" that both WLED and WLEDpixelBus can depend on. (wled::colors ?? ) Otherwise we should use the definitions for colors.h directly. (It makes sense in its way: one task performed here is to map between WLED's internal color format to the output pixel format.)
• The split between BusDigital and PixelBus is costly -- calls like setPixelColor invoke two vtable lookups. (And a third dynamic dispatch for the color encoding -- switch and virtual are conceptually similar.) I don't have an immediate suggestion; this is mostly a performance tuning concern and we don't necessarily have to address it up front. From an API perspective, the key change would be moving to a factory function (eg replacing new BusDigital(...) to makeDigitalBus(...)) -- this would let the PixelBus system someday return a fully qualified Bus type with fewer dynamic dispatches.
• CustomBusConfig -- Good call to factor this separately from the common BusConfig, but directly embedding it in BusConfig gives up all the advantages. I suggest using a type-erased bus-specific-settings slot in the BusConfig level (eg. instead of CustomBusConfig custom, do std::shared_ptr<void> customBusSettings or the like), and then cast it to the bus-specific struct during construction. This lets each Bus implement its own custom settings struct and doesn't require sharing the type outside of the local bus implementation and local bus settings parser. (The next step will be to factor the config parser on a per-TYPE_ basis, leveraging getLEDTypes(), but that's a plan for another day.)

[DedeHai]

Thanks for taking a dive.

I'm finding it challenging to review this properly - I just keep exploding in a hundred nitpicks. It's hard to know where to start and I feel bad pushing suggestions in dribs and drabs.

Fully understand, its a major overhaul and a "diamond in the rough", I also did not start out with a full concept but kept adding and replacing things (as you can see in the PR having over 200 commits). Each time I read through the code I find things that could be done better or more clearly but let's do it in increments, its just too much code to have a clean overview and its easy to get lost in details.

Bigger things I've noticed so far:

• WLEDpixelBus.h has duplicate copies of code from colors.h. This component already isn't isolated from WLED internals (it's importing const.h for WLED's type mapping).

This was actually intentional - I did try to keep current code & structure intact as much as possible, also to not change things and loose backwards compatibility. I am not saying it's the way it has to be in the end.

If we want to maintain the conceit that this is an independent library, pull out colors.h to its own "library" that both WLED and WLEDpixelBus can depend on. (wled::colors ?? ) Otherwise we should use the definitions for colors.h directly. (It makes sense in its way: one task performed here is to map between WLED's internal color format to the output pixel format.)

I did not build it up to be an independent library - but ultimately that should be the goal. I did focus on getting the drivers to work and somehow integrate them and less on making it as clean - as I am sure you can tell ;)

• The split between BusDigital and PixelBus is costly -- calls like setPixelColor invoke two vtable lookups. (And a third dynamic dispatch for the color encoding -- switch and virtual are conceptually similar.) I don't have an immediate suggestion; this is mostly a performance tuning concern and we don't necessarily have to address it up front. From an API perspective, the key change would be moving to a factory function (eg replacing new BusDigital(...) to makeDigitalBus(...)) -- this would let the PixelBus system someday return a fully qualified Bus type with fewer dynamic dispatches.

Agreed, the point from above applies here as well: I kept the old structure to not get tangled up and keep my focus on the pixel-pushing driver. I am not sure on where to make the split i.e. should the WLEDpixelBus replace bus digital or be more of an independent library.
edit:
it is still (way) faster than NPB - apart from the lower level driver being faster I think the main speedup is from not doing as many color conversions, those are expensive, the color order lookup is also much faster.

• CustomBusConfig -- Good call to factor this separately from the common BusConfig, but directly embedding it in BusConfig gives up all the advantages. I suggest using a type-erased bus-specific-settings slot in the BusConfig level (eg. instead of CustomBusConfig custom, do std::shared_ptr<void> customBusSettings or the like), and then cast it to the bus-specific struct during construction. This lets each Bus implement its own custom settings struct and doesn't require sharing the type outside of the local bus implementation and local bus settings parser. (The next step will be to factor the config parser on a per-TYPE_ basis, leveraging getLEDTypes(), but that's a plan for another day.)

sounds like a good idea, I have no real grasp on that concept though - it's a bit above my every-day C++ knowledge but I am sure I can learn that abstraction level.
I at one point had custom bus config as it's own config but then also wanted to be able to modify "standard" configs, i.e. take any of the supported bus formats and reorder the output format. If I understand correctly, your suggestion gets the best of both worlds.

Maybe we should have a call to discuss some of this?

[DedeHai]

@willmmiles one issue that remains is the default color order (I do have a solution for color order override but need to review and test it). I had an AI summarise the color orders used by the NPB buses in the original buswrapper. Its pretty inconsistent, assuming the summary is correct.
The question is: do we add a translation table so the new driver uses the same default color orders for the different LED types? It is not pretty carrying them over but otherwise users updating will have their color order changed, causing frustration.

This is what the AI spit out, I checked some of the values and the ones I checked were assessed correctly:

AI START

 

WLED’s UI/config default is always COL_ORDER_GRB unless overridden by a color-order mapping. This is defined in bus_manager.h:472 and const.h:394-399. With the legacy driver, PolyBus first applies co, then NeoPixelBus applies the feature’s native order in bus_wrapper.h:791-813.

WLED type	NeoPixelBus feature	Native order with default co=GRB
TYPE_WS2812_RGB	NeoGrbFeature	GRB
TYPE_WS2812_1CH_X3	NeoGrbFeature	GRB carrier; custom single-channel mapping
TYPE_WS2812_WWA	NeoGrbFeature	GRB carrier; special WWA packing
TYPE_WS2811_400KHZ	NeoGrbFeature	GRB
TYPE_TM1829	NeoBrgFeature	BRG
TYPE_UCS8903	NeoRgb48Feature	RGB, 16-bit words
TYPE_UCS8904	NeoRgbw64Feature	RGBW, 16-bit words
TYPE_SK6812_RGBW	NeoGrbwFeature	GRBW
TYPE_TM1814	NeoWrgbTm1814Feature	WRGB
TYPE_APA106 ESP32	NeoGrbFeature	GRB
TYPE_APA106 ESP8266 UART	NeoRbgFeature	RBG
TYPE_APA106 ESP8266 DMA/BitBang	NeoGrbFeature	GRB
TYPE_FW1906	NeoGrbcwxFeature	G,R,B,CW,WW,0
TYPE_WS2805	NeoGrbwwFeature	G,R,B,W1,W2 = GRB,WW,CW
TYPE_TM1914 ESP32	NeoGrbTm1914Feature	GRB
TYPE_TM1914 ESP8266	NeoRgbTm1914Feature	RGB
TYPE_SM16825	NeoRgbcwSm16825eFeature	RGB,CW,WW, 16-bit words
TYPE_WS2801	NeoRbgFeature	RBG
TYPE_APA102	DotStarBgrFeature	BGR
TYPE_LPD8806	Lpd8806GrbFeature	GRB
TYPE_LPD6803	Lpd6803GrbFeature	GRB
TYPE_P9813	P9813BgrFeature	BGR

The exact feature selections are in bus_wrapper.h:159-209 and bus_wrapper.h:281-315. The NeoPixelBus feature definitions confirm the channel sequences, including NeoBrgFeature, NeoGrbcwxFeature, and the SM16825 aliases.

TYPE_WS2812_1CH and TYPE_GS8608 are declared but unused/unmapped in this legacy dispatch. Analog, network, on/off, and HUB75 types do not use NeoPixelBus color-order features.

For the new driver, the existing TODO is accurate: WLEDpixelBus.cpp needs chip-specific native-order handling. Pay special attention to TM1829, APA106, TM1914, FW1906, WS2805, SM16825, WS2801, APA102, and P9813.

AI END

[willmmiles]

Hmm, that is tricky.

Probably what I'd suggest is to change the config schema (though I think we should consider that anyways). If the new (and accurate) schema is present, use it; if only the old keys are present, adapt them using a compatibility table.

If it were up to me I'd get rid of the TYPE numeric mapping entirely in favor of driver names, and have the common "digital" api always accept raw timings and explicit color order (eg. a string "RGBWC0" or the like -- defining the whole mapping order). The predefined types/timings table would be left to the UI.

[DedeHai]

I like the idea. The LED types could be put into a LEDtypes.json file and served as a gzip to the UI to read, just like we do for the htm pages. New/custom types could live in userLEDtypes.json on the file system.