Use with Rust

[joshbenaron]

Is there a way to implement this in Rust? Or is this already done? I just want to do basic operations like add to a queue etc. Is there a rough high level api for key structures etc and what I need to store in order to do this?

[ccollie]

bullmq is distributed as a library, so no api is included, though its easy enough to set up a server with express/fastify. Check out bull-monitor for how they do this. They're oriented towards the UI but as part of the process they both provide an api for the frontend to use.

[ccollie]

BTW. Your use case is precisely one of the reasons I'm developing a graphql server on top of bullmq

[paulmikulskis]

thanks dope!

[joshbenaron]

Well - I might not follow. But what I want to do is use a Rust http server. For each request I add a job to the queue. I simply want to add to the queue from the Rust server

[joshbenaron]

Surely adding an item to the queue is just adding specific key/values to Redis?

[ccollie]

You can certainly add jobs via direct access to redis, but that's incredibly brittle. If you're really insistent and are comfortable in calling redis from rust, check out

https://github.com/taskforcesh/bullmq/blob/master/src/commands/addJob-9.lua

Calling that should get you what you want. My question though is how are your workers implemented ? I'm unaware of any non-node projects using bullmq. If you're using node workers anyway, then why not use it to add jobs ?

[joshbenaron]

So the http server is rust - and yeah the workers are nodejs. I want high performance for the http server

[joshbenaron]

incredibly brittle

In what sense? @ccollie

[ccollie]

Its brittle because you're sneaking around the back of the api. Bullmq is implemented both in node and as scripts in redis, so it's important to know how they interact.

As far as the server being in rust, I honestly wouldn't worry about it, especially since all you'd be doing is pushing data to redis, which is IO bound in any event. Bull also has methods to do bulk pushes if you need to squeeze out extra performance.

If you really need speed, I'd implement the workers in rust, since bull allows for manual queue handling.

[ccollie]

To be more specific, if all you're doing in the http server is pushing data to redis, the performance differential would not justify the introduction of a new technology, IMO. Node is perfectly at home with IO bound processing, whereas rust could definitely be a benefit for workers if most of your queue tasks are CPU bound.

[joshbenaron]

I meant I want performance for the server in general. One route adds to the queue. i.e. there is some CPU bound stuff that the server does

[joshbenaron]

How would I go about manually processing jobs from Rust?

[ccollie]

I guess if you have other reasons for having a rust based server then that's understandable.

r.e. Manual Processing:
Looking at the docs here:
https://docs.bullmq.io/patterns/manually-fetching-jobs

I may have been overly optimistic, since this uses the bull js library itself. It seems like you would have to replicate the worker js code in rust and follow the directions in the link.

[joshbenaron]

Cool. I'll start with the adding first. Long term I'll make this into a nice Rust lib for people to use

[ccravens]

@joshbenaron curious what performance concerns you have with NodeJS? Typically performance issues are not due to the programming language you choose but more so how the processing is implemented.

Also, if you're REALLY worried about performance have you considered another stack like ZeroMQ (https://github.com/erickt/rust-zmq) with Cap'n Proto (https://capnproto.org/). Although I'm not fully convinced that you'd likely see much in the line of performance gains to be honest.

[joshbenaron]

@ccollie have you got a repo with your work? I'm working on a similar problem.

[ccollie]

@joshbenaron i'm still not through writing the lib, so the node interface is still a bit off, but check out https://github.com/Brooooooklyn . He has a few projects creating npm packages out of rust libs.

[joshbenaron]

any update here @ccollie ?

[ccollie]

Not really @joshbenaron. I'm in a new position that has taken me away from BullMQ, though I've spent the last 8 or so months working on an opensource Rust project. I recall seeing a few projects on crates.io implementing workers on top of redis, so maybe you can look there for inspiration. There's also rustrepo.com.

If you have a project that I where I can pitch in periodically, I'd be happy to lend a hand, though my time is constrained at the moment.

[opeolluwa]

Could you please point out a few of these libraries, I need a performant Message Queue lib to connect a node and Rust application

[vstepanyuk]

Can you guide me to the documentation or code that would help me to post a flow (parent/children jobs) from Rust?

[manast]

@vstepanyuk you will basically need to port this class to Rust:
https://github.com/taskforcesh/bullmq/blob/master/src/classes/flow-producer.ts
You can load the required .lua scripts from rawScripts as the python implementation is doing:
https://github.com/taskforcesh/bullmq/blob/master/python/bullmq/scripts.py