examples: Implement drivers for microwindows

[Acfboy]

Hello the community! Here is the work I have done for the first half of GSoC, now complete and ready for review.

Work Completed

The GDI-level Microwindows core, drivers, and mwdemo are functional on QEMU x86-64 and the simulator.

Build System Integration

• Based on the initial proof-of-concept branch by @ppisa, the Microwindows core (engine, drivers, precompiled bitmap fonts) has been successfully compiled into the NuttX apps library.
• Uses MWCONFIG_FILE="mwconfig.nuttx" to inject NuttX-specific configuration into Microwindows without touching upstream headers.
• Image resources used by mwdemo.c are compiled into binary using the convbmp tool during the build.

NuttX Hardware Drivers

Framebuffer driver (scr_nuttx.c), keyboard drivers (kbd_nuttx_event.c for event-mode and kbd_nuttx_raw.c for raw byte-stream), mouse driver (mou_nuttx_mouse.c), and touchscreen driver (mou_nuttx_ts.c) have been implemented and submitted to upstream Microwindows mainline ghaerr/microwindows#193. The graphics/microwindows/Makefile selects the appropriate drivers via ARCH=NUTTX and Kconfig-driven KEYBOARD/MOUSE variables.

Kconfig Options

Flexible Kconfig for keyboard driver: event-mode (/dev/kbd), raw byte-stream via kbd_codec (/dev/kbda), none, or custom. For mouse/touchscreen: relative mouse (/dev/mouse0), touchscreen (/dev/input0), none, or custom. Custom options allow BSP/app-level overrides without modifying the Microwindows build logic.

Demo Application

Ported mwdemo.c (a complex Windows-like demo) into a standalone NuttX example application. The demo runs successfully on both qemu-intel64:mw and the NuttX simulator (sim:mw).

I have also tested the demo with #define CONTROL 1 enabled in mwdemo.c, which represents a more complex use case.

Upstream Bug Fixes

Several issues were discovered, analysed, and most of them already fixed:

Issue	Root Cause	Fix
Animation freezing in mwdemo	MwSelect() blocked forever when a timer had already expired, because the code assumed no timer existed when GdGetNextTimeout returned false.	Patch by ppisa, correctly handle the case where timeout == 0 and return immediately.
Cursor leaving a blue ghost image	#if defined(ELKS) in mwtypes.h forced MWPIXELVALHW to unsigned char even when ELKS=0 (intended to disable it). This truncated 32-bit pixel values read from the framebuffer.	Changed to #if ELKS
System freeze when dragging a window caption	During drag, MwSelect switches to polling (select(…, timeout=0)). The USB input thread had a lower priority (50) than the application (100), so it never got CPU time to generate events.	Short-term: adjusted thread priorities in defconfigs. A permanent solution in Microwindows may be discussed later.
Mouse drift in simulator after dragging	TOUCH_POS_VALID flag was not being checked in the touchscreen driver, causing uninitialized coordinate values to be used.	Fixed in mou_nuttx.c by checking TOUCH_POS_VALID before using touch coordinates.
KEYCODE_xxx and ASCII range conflict	NuttX's KEYCODE_xxx macros (e.g., KEYCODE_PAUSE = 0x20) overlap with ASCII characters, making it impossible to distinguish special keys from normal input in the event-mode keyboard driver.	Still under discussions apache/nuttx#19527

How to Reproduce (QEMU / Simulator)

qemu-intel64:mw

make distclean
./tools/configure.sh qemu-intel64:mw
make -j$(nproc)

Then create a bootable disk following the documentation and run:

qemu-system-x86_64 -cpu host -enable-kvm -m 2G \
        -cdrom boot_mw.iso \
        -serial stdio -monitor none -no-reboot -device qemu-xhci -device usb-kbd -device usb-mouse

sim:mw

make distclean
./tools/configure.sh sim:mw
make -j$(nproc)
./nuttx

Notice

• You can set #define CONTROL 1 manually in mwdemo.c to test more complex case.
• Most of the nxstyle problems in mwdemo.c are fixed, except the names of Win32 API.
• A draft document for microwindows is at Acfboy/nuttx@f4d9316
• The Nano-X X11 supprot is still work in progress.

[ppisa]

Thanks to @Acfboy for the work and this draft request for discussion and @ghaerr for consultations and support on Microwidows side.

I am leaving for four days now, but there are some my thoughts, the NuttX Microwidows screen, mouse and keyboard drivers should be submitted with appropriate build enable options to the mainline Microwindows https://github.com/ghaerr/microwindows before the final nuttx-apps pull request.

[acassis]

@Acfboy please fix the issues on mwdemo:

[simbit18]

Hi @Acfboy , could you also add support for building with CMake?

[Acfboy]

Hi @Acfboy , could you also add support for building with CMake?

Thank you for the reminder. I will add CMake support later.

[acassis]

@Acfboy could you spend some time fixing and improving this PR? It still as Draft. It is important to get it merged and let more people to test and review it. Also please remember to include a proper nuttx/Documentation about nanox/microwindows support on NuttX

[Acfboy]

Hi @acassis , thank for the reminder. I agree I should submit the previous work as a proper PR for the community to review. However, for it to be a proper PR, I think I need to merge the existing NuttX drivers into the Microwindows mainline first. I apologize for postponing this part of the work earlier. I will submit a PR for upstream MW review soon.

[ppisa]

Yes, this what I have proposed and expect as well.

You should set something like SCREEN=NUTTX or SCREEN=NUTTX_FB in the local graphics/microwindows/Makefile and then Microwindows with merged code should add correct driver sources when you include microwindows/src/drivers/Objects.rules.

Same for the MOUSE and KEYBOARD. The question is if the mouse should be a single driver or one for relative motion mouse device and other for touchscreen.

As for the keyboard, there can be space to think how to map it the best way. The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII. Microwindows works with MWKEY and MWSCANCODE which aligns with concept of real X11 where the keyboards provide "position" of pressed key by scancode and it can be remapped to symbol even latter in application. But usually MWKEY code is used and its definitions are provided in src/include/mwtypes.h.

You provide translation in translate_keycode when keyboard is in g_kbd_mode set to KBD_MODE_EVENT but there is heuristics during open. I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name. So it is necessary to think a little what is the best option there.

You should try even if Nano-X server and clients can be run on NuttX. I do not see X11 graphics option as the main priority, but it would prove that even complex setup can be build and check if it is stable on NuttX.

[Acfboy]

The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII.

Hi, @ppisa . I have a question about this statement. It seems to me that the ranges of the KEYCODE_xxx macros and ASCII are conflicting, which makes it impossible for my driver to distinguish whether it's a character or a special key. In my sim:mw setup, this results in being unable to type a space.

I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name.

Two drivers makes the code cleaner. But specify the device path/name via Kconfig risks inconsistency with the real device. Alternatively, if Microwindows auto-selects the device based on NuttX config, then we have to keep both sides in sync — which adds maintenance overhead.

You should try even if Nano-X server and clients can be run on NuttX.

Okay, I'll try server and client of NanoX.

[ppisa]

The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII.

Hi, @ppisa . I have a question about this statement. It seems to me that the ranges of the KEYCODE_xxx macros and ASCII are conflicting, which makes it impossible for my driver to distinguish whether it's a character or a special key. In my sim:mw setup, this results in being unable to type a space.

Yes, you are right, I expected that for KEYCODE_NORMAL the ASCII value is returned in another field of the event structure, but there is only

struct keyboard_event_s
{
  uint32_t type;
  uint32_t code;
};

where type is KEYBOARD_PRESS or KEYBOARD_RELEASE. But into code sent to keyboard_event() is ASCII code for example in g_km_keymap in STM32 matrix keyboard but some other drivers call keyboard_event() even with X11 encoded KeySim (arch/sim/src/sim/posix/sim_x11eventloop.c) and some with keys translated to KEYCODE_xxx where is no space for ASCII.

So this seems to be call for priority issue.

@acassis @gregory-nutt Please, do you have some some insight what is right?

The simple fix is to push KEYCODE_FWDDEL in include/nuttx/input/kbd_codec.h to be above ASCII, for example 128 or 129. But may it be it can break some targets where it would result in some large arrays, etc. When keyboards with interpreted national symbols are used then it would worth to push special codes even further after basic unicode...

I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name.

Two drivers makes the code cleaner. But specify the device path/name via Kconfig risks inconsistency with the real device. Alternatively, if Microwindows auto-selects the device based on NuttX config, then we have to keep both sides in sync — which adds maintenance overhead.

Yes but NuttX is based on configuration and it is better to fail then to do lot of testing of different device names and even deciding if it is raw driver or event based. But in general it seems that there is pace for some discussion and making KBD simple on the NuttX side.

You should try even if Nano-X server and clients can be run on NuttX.

Okay, I'll try server and client of NanoX.

Thanks.

[acassis]

The NnuttX provides its KEYCODE_xxx macros in kbd_codec.h for special keys. But only for these and rest is expected to be ASCII.

Hi, @ppisa . I have a question about this statement. It seems to me that the ranges of the KEYCODE_xxx macros and ASCII are conflicting, which makes it impossible for my driver to distinguish whether it's a character or a special key. In my sim:mw setup, this results in being unable to type a space.

Yes, you are right, I expected that for KEYCODE_NORMAL the ASCII value is returned in another field of the event structure, but there is only

struct keyboard_event_s
{
  uint32_t type;
  uint32_t code;
};

where type is KEYBOARD_PRESS or KEYBOARD_RELEASE. But into code sent to keyboard_event() is ASCII code for example in g_km_keymap in STM32 matrix keyboard but some other drivers call keyboard_event() even with X11 encoded KeySim (arch/sim/src/sim/posix/sim_x11eventloop.c) and some with keys translated to KEYCODE_xxx where is no space for ASCII.

So this seems to be call for priority issue.

@acassis @gregory-nutt Please, do you have some some insight what is right?

The simple fix is to push KEYCODE_FWDDEL in include/nuttx/input/kbd_codec.h to be above ASCII, for example 128 or 129. But may it be it can break some targets where it would result in some large arrays, etc. When keyboards with interpreted national symbols are used then it would worth to push special codes even further after basic unicode...

I think that there could be two drivers, one for event type and another for raw keyboards. This could be configured through Kconfig, same would be useful for device name.

Two drivers makes the code cleaner. But specify the device path/name via Kconfig risks inconsistency with the real device. Alternatively, if Microwindows auto-selects the device based on NuttX config, then we have to keep both sides in sync — which adds maintenance overhead.

Yes but NuttX is based on configuration and it is better to fail then to do lot of testing of different device names and even deciding if it is raw driver or event based. But in general it seems that there is pace for some discussion and making KBD simple on the NuttX side.

You should try even if Nano-X server and clients can be run on NuttX.

Okay, I'll try server and client of NanoX.

Thanks.

Hi @ppisa I don't know the right approach to keyboard input keys symbols. Actually @linguini1 faced similar issue when porting Doom to NuttX. Unfortunately NuttX doesn't have a standard keyboard symbols

[linguini1]

One option suggested by @cederom was to adopt the EVDEV codec: https://en.wikipedia.org/wiki/Evdev

[ppisa]

Hi @ppisa I don't know the right approach to keyboard input keys symbols. Actually @linguini1 faced similar issue when porting Doom to NuttX. Unfortunately NuttX doesn't have a standard keyboard symbols

@linguini1 and @acassis, I propose simple fix include/nuttx/input/kbd_codec.h

-  KEYCODE_FWDDEL,
+  KEYCODE_FWDDEL = 128,     

Or even larger shift to something like 0xF800 as used on Microwindows.

This would solve problem with overlap of letters and other ASCII codes with special keys.

I can open issue for this. The question is if there is something, some other keyboard driver or application, which breaks by this shift of KEYCODE_xxx. It would be problem if somebody uses the codes as index into some array etc... 128 or 129 shift is not so big so it less expensive if somebody uses arrays for some key code transformation.

I think that that this is reasonable solution for NuttX and current model.

[linguini1]

I agree, and this was a solution I considered but ultimately did not have enough time to verify if it would break other things in the kernel. I think that is at least a good interim solution which is less invasive.

[ghaerr]

Or even larger shift to something like 0xF800 as used on Microwindows.

Looking ahead at Unicode, Microwindows moved its key code base to F800 because that belongs to the Unicode BMP Private Use Area. This won't overlap other Unicode values if/when NuttX applications move to Unicode.

It would be problem if somebody uses the codes as index into some array etc... 128 or 129 shift is not so big

Possibly good idea, but pushes off actively looking through source for problems now, and not having to change them again perhaps later. FWIW, myself knowing very little about NuttX, if a small shift is thought better for now, choosing a base of 256 at least allows space for using the upper half of various 256-byte multilingual code pages for non-US (e.g. European) accented characters, etc. OTOH, if keycodes are stored in byte arrays, then 128/129 would likely be mandated.

[Acfboy]

Hi @xiaoxiang781216 @acassis thanks for the thorough review! I've addressed all your feedback and fixed the non-standard commit messages flagged by checkpatch.

[xiaoxiang781216]

Hi @xiaoxiang781216 @acassis thanks for the thorough review! I've addressed all your feedback and fixed the non-standard commit messages flagged by checkpatch.

please merge your temp change.

[ppisa]

@Acfboy please, squash, merge commits as @xiaoxiang781216 suggests into single commit or some small logical set of incremental commits, there is no use for keeping the new component development history, you ca keep it on some branch of your repository as work progress documentation. So for logical commits series, I can imagine one commit which introduces Microwindows with basic Kconfig then another one which adds some demos, Kconfig options, etc. But the code has to be clean, adhere NutttX style and requirements and build after each incremental commit.

As for the decision, which defines should go into Makefile and which to Make.defs, I think that only these defines which can be used by potential applications directly or through public Microwindows header files should be present in Make.defs, Sofor example, if SCREEN_PIXTYPE is made configurable through Kconfig and then used by header files, then it should be defined in Make.defs, same i required for additional include locations. On the other hand defines provided only to choose correct behavior during library build i.e. through Objects.rules should be enough to define only in related NuttX Makefile. But I think that current state is mostly aligned with these requirements.

[Acfboy]

Thanks for pointing this out, @ppisa @xiaoxiang781216 . I've reorganized the changes into two commits.

[Acfboy]

Thanks @xiaoxiang781216 for the very thorough review, and @ppisa for the clarification and discussion. I have updated the code under your guidance.

Regarding whether all include paths in Make.defs should be moved to the Makefile, I agree with @ppisa that this would require more changes to the build system, so I haven't made that change for now.

[xiaoxiang781216]

Thanks @xiaoxiang781216 for the very thorough review, and @ppisa for the clarification and discussion. I have updated the code under your guidance.

Regarding whether all include paths in Make.defs should be moved to the Makefile, I agree with @ppisa that this would require more changes to the build system, so I haven't made that change for now.

@Acfboy please always address the comment in the origin patch instead creating new one.

[ghaerr]

Generally, I see as the best solution do not need for these to be defined by build system to zero.

On the Microwindows side, for various reasons the idiom of #if RTEMS etc is used a lot more than #ifdef RTEMS, mostly so that the same mechanism can be used to turn options on or off without changing an #ifdef to an #if. However, it was not the intent to force developers to find all cases of #if SYM and pass -DSYM=0 everywhere. Thus, I do not agree the best solution is to set otherwise undefined symbols to 0.

It seems this may be a problem because of the use of -Wundef in NuttX?

There is one exception
#if __ECOS

This meas that if this header file is included without preceding mwconfig.h include, then there could be a problem.

The ECOS port is quite old, and probably doesn't follow some of the more recent ideas of keeping platform build-specific options in a separate configuration file. This could be fixed, but I hesitate to change code that I or others can't easily test in the main repo.

[ppisa]

Generally, I see as the best solution do not need for these to be defined by build system to zero.

On the Microwindows side, for various reasons the idiom of #if RTEMS etc is used a lot more than #ifdef RTEMS, mostly so that the same mechanism can be used to turn options on or off without changing an #ifdef to an #if. However, it was not the intent to force developers to find all cases of #if SYM and pass -DSYM=0 everywhere. Thus, I do not agree the best solution is to set otherwise undefined symbols to 0.

OK, I have tried to add checks for defines presence in Microwindows code. I agree that there is some added complexity. I have repeated some defines sequences because with checks for define presence the conditions testing multiple target OS alternatives are hard to write such that warning is prevented

ghaerr/microwindows#194

[Acfboy]

Thank you for your careful review, detailed discussion, and guidance. I have made the revisions.

[xiaoxiang781216]

@acassis @ppisa is ready for merging.

[ppisa]

It seems that I am late to this party/iteration as the work is merged.

@Acfboy has done sound work and as the Microwidows are fully optional component and the main work on NuttX drivers API related code is already integrated into mainline Microwidows, then merge can be base for next @Acfboy work and good opportunity for others to see the integration as preview. On the other hand I do not consider this as production ready/final version.

@Acfboy has even prepared documentation and configs with examples configurations qemu-intel64:mw and sim:mw on related NuttX fork https://github.com/Acfboy/nuttx/commits/add-microwindows/

It would worth to be integrated through pull-request as well. But I would like to discuss some more options for example configurations. I would suggest to start the Microwidows demo through NSH to allow console access into running system for debugging, etc. But these are details which will be sorted out.

As for the maim Microwindows integration, there worth to be done mone polishment to find reasonable way how to resolve warnings, solve apache/nuttx#19527 (hope to find time to propose patch today). Then the test of more complex demos and applications based for example on X11 API should be tested. We will se where we find problems or some NuttX specific variants would be required... But as both Microwindows and NuttX use POSIX as main runtime model, it should be non-intrusive on both sides.

I would be happy if components headers export for NuttX make export is solve in future which is useful not only for Microwidows but more general way.

[xiaoxiang781216]

It seems that I am late to this party/iteration as the work is merged.

@Acfboy has done sound work and as the Microwidows are fully optional component and the main work on NuttX drivers API related code is already integrated into mainline Microwidows, then merge can be base for next @Acfboy work and good opportunity for others to see the integration as preview. On the other hand I do not consider this as production ready/final version.

@Acfboy has even prepared documentation and configs with examples configurations qemu-intel64:mw and sim:mw on related NuttX fork https://github.com/Acfboy/nuttx/commits/add-microwindows/

It would worth to be integrated through pull-request as well. But I would like to discuss some more options for example configurations. I would suggest to start the Microwidows demo through NSH to allow console access into running system for debugging, etc. But these are details which will be sorted out.

As for the maim Microwindows integration, there worth to be done mone polishment to find reasonable way how to resolve warnings, solve apache/nuttx#19527 (hope to find time to propose patch today). Then the test of more complex demos and applications based for example on X11 API should be tested. We will se where we find problems or some NuttX specific variants would be required... But as both Microwindows and NuttX use POSIX as main runtime model, it should be non-intrusive on both sides.

I would be happy if components headers export for NuttX make export is solve in future which is useful not only for Microwidows but more general way.

Thanks for the detailed summary and future plan. Do you have Nano X11 progress @ppisa ?

[ppisa]

Thanks for the detailed summary and future plan. Do you have Nano X11 progress @ppisa?

This is more to @Acfboy , I try to help where I have knowledge and as I find a time. I expect that it can be relatively straightforward. But who know. We need UNIX sockets or other local connection. But it should be available on NuttX.

[Acfboy]

Yes, I'm working on porting the nano-X X11 API and will likely submit a new PR soon. After that, I'll do more thorough testing, including on real hardware, and address all warnings to make it production-ready.