Challenges with the STA server message loop and firing events

[bennyrowland]

I use comtypes for making several local servers to provide plugins for another application. I have a lot of challenges working with the MessageLoop class (

comtypes/comtypes/messageloop.py

Line 18 in 76c3542

class _MessageLoop(object):

) for single threaded apartments, specifically for firing events.

The MessageLoop basically blocks on GetMessage() all the time, until a message comes in, which then gets handled, and we go back to blocking on GetMessage() again. This means that it is basically impossible to do anything on the main thread, except inside the handler for a COM method. My server design requires the client calling methods which initiate long running calculations on the server but return immediately to avoid the client blocking on the method call.

It is possible to partially relieve this problem by adding a second thread in the server - the main thread gets the method from COM, puts it in a queue for the second thread to pick up when ready, then returns immediately. The issue then comes when the second thread wants to signal something back to the client via an event. Because of the STA model, only the main thread is able to call Fire_Event() to deliver the event to the client, but because that is blocked by the call to GetMessage(), we cannot actually fire the event (unless we wait for an incoming method call from the client and use that to fire any pending events).

Running in MTA mode allows firing the event from the second thread but this has other consequences for thread safety and I would rather avoid it if possible. Therefore I am looking for suggestions on how we might make firing events from servers a more straightforward task. At the moment my solution is to have an intermediate "event-firing" server, which provides an interface of methods which can be called which immediately synchronously fire the corresponding event. An instance of this server is created by the client (which connects to its events) and passed to the primary server, which marshals it over to the secondary thread (via the GIT) which is then able to call its methods and fire events on the client. This works, but is obviously a sub-optimal design.

What I (currently) think would be nicest would be some way of getting in to the MessageLoop internally without a COM method call coming in. I think it would be really cool if we could interface the loop with cooperative multitasking like asyncio, then it would be easy to run other tasks on the main thread interspersed with handling COM method calls. A secondary option might be to decouple Fire_Event so that it can be called in a thread-safe manner - when a secondary thread calls Fire_Event it could put the event in a queue, then post a message in the message loop that would trigger a handler on the main thread to actually fire queued events.

I would also be very happy to hear any other suggestions or alternative approaches to solving this problem. Thanks for your input.

[junkmd]

I would like to support your challenge.

I have mostly used this package on the client side, so it's reassuring to know that there are people in the community who are knowledgeable about server-side use cases and implementations.

I take this issue as a feature request for comtypes.

As a maintainer, my concerns are:

• Whether it will break backward compatibility
• Whether it will complicate other parts besides the message loop to maintain backward compatibility
• How to test it if implemented

I recognized that there are no tests for the message loop in this package.
If we are to add or change functionality related to the message loop, having tests that ensure the current behavior beforehand would allow me as a maintainer to consider the proposal with peace of mind.
Testing event firing is challenging.
I am developing the frontend with TypeScript and React for a closed-source project, and I always struggle with inspecting event firing and its side effects when involved in that task.

If there is at least one other contributor willing to cooperate on this challenge, I would like to pin this issue.

[bennyrowland]

@junkmd, thanks for your support. I agree that further test support in this area is probably a good idea, but the bigger issue to adding this feature is my lack of knowledge about how the underlying Windows API works, quite simply I have no idea how we could approach solving this problem. Unless there is somebody in the community who has a deeper knowledge of these details than I do, I don't think we can make any forward progress. I am happy to put in the work if someone can give me the necessary pointers, but for now I am stuck.