(py)zmq/pub : Is It Possible To Call Connect() Then Send() Immediately And Do Not Lose The Message?

With this code, I always lose the message : def publish(frontend_url, message): context = zmq.Context() socket = context.socket(zmq.PUB) socket.connect(frontend_url)

Solution 1:

You can't do this exactly, but there may be other solutions that would solve your problem.

Why are you using PUB/SUB sockets? The nature of pub/sub is more suited to long-running sockets, and typically you will bind() on the PUB socket and connect on the SUB socket. What you're doing here, spinning up a socket to send one message, presumably to a "server" of some sort, doesn't really fit the PUB/SUB paradigm very well.

If you instead choose some variation of REQ or DEALER to REP or ROUTER, then things might go smoother for you. A REQ socket will hold a message until its pair is ready to receive it. If you don't particularly care about the response from the "server", then you can just discard it.

Is there any particular reason you aren't just leaving the socket open, instead of building a whole new context and socket, and re-connecting each time you want to send a message? I can think of some limited scenarios where this might be the preferred behavior, but generally it's a better idea to just leave the socket up. If you wanted to stick with PUB/SUB, then just spin the socket up at the start of your app, sleep some safe period of time that covers any reasonable latency scenario, and then start sending your messages without worrying about re-connecting every time. If you'll leave this socket up for long periods of time without any new messages you'll probably want to use heart-beating to make sure the connection stays open.

Solution 2:

From the ZMQ Guide:

There is one more important thing to know about PUB-SUB sockets: you do not know precisely when a subscriber starts to get messages. Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends. This is because as the subscriber connects to the publisher (something that takes a small but non-zero time), the publisher may already be sending messages out.

Solution 3:

Many posts here start with:

"I used .PUB/.SUB and it did not the job I wanted it to do ... Anyone here, do help me make it work like I think it shall work out of the box."

This approach does not work in real world, the less in distributed systems design, the poorer in systems, where near-real-time scheduling and/or tight resources-management is simply un-avoid-able.

Inter-process / inter-platform messaging is not "just another" simple-line-of-code (SLOC)

# A sample demo-code snippet          # Issues the demo-code has left to be resolved#------------------------------------ #------------------------------------------------defpublish( frontend_url, message ): # what is a benefit of a per-call OneStopPUBLISH function?
    context = zmq.Context()           # .Context() has to be .Terminate()-d (!)
    socket = context.socket(zmq.PUB)  # is this indeed "a disposable" for each call?
    socket.connect(frontend_url)      # what transport-class used for .connect()/.bind()?
    time.sleep(0.1)                   # wait for the connection to be established 
    socket.send(message)              # ^ has no control over low-level "connection" handshaking

Anybody may draft a few one-liners and put a decent effort ( own or community outsourced ) to make it finally work ( at least somehow ).

However this is a field of vast capabilities and as such requires a bit of reshaping one's mind to allow its potential to become unlocked and fully utilised.

Sketching a need for a good solution but with wrong grounds or mis-understood SLOC-s ( be it copy/paste-d or not ) typically does not yield anything reasonable for the near, the less for the farther future.

Messaging simply introduces a new paradigm -- a new Macro-COSMOS -- of building automation in wider scale - surprisingly, your (deterministic) code becomes a member of a more complex set of Finite State Automata ( FSA ), that - not so surprisingly, as we intend to do some "MESSAGING" - speak among each other.

For that, there needs to be some [local-resource-management], some "outer" [transport], some "formal behaviour model etiquette" ( not to shout one over another ) [communication-primitive].

This is typically in-built into ZeroMQ, nanomsg and other libraries.

However, there are two important things that remain hidden.

• The micro-cosmos of how the things work internally ( many, if not all, attempts to tweak this, instead of making one's best to make proper use of it, are typically waste of time )
• The macro-cosmos of how to orchestrate a non-trivial herd of otherwise trivial elements [communication-primitives] into a ROBUST, SCALEABLE messaging ARCHITECTURE, that co-operates across process/localhost/network boundaries and that meets the overall design needs.

Failure to understand the distance between these two worlds typically causes a poor use of the greatest strengths we have received pre-cooked in the messaging libraries.

Simply the best thing to do is to forget the one-liner tweaking approaches. It is not productive.

Understanding the global view first, allows you to harness the powers that will work best for your good to meet your goals.

Why it is so complex?

( courtesy nanomsg.org )

Any non-trivial system is complex. Both in TimeDOMAIN and in ResourcesDOMAIN. The more, if one strives to create a stable, smart, High-performance, Low-latency, transport-class-agnostic Universal Communication Framework.

The good news is, this has been already elaborated and in-built into the micro-cosmos architecture.

The bad news is, this does not solve your needs right from the box ( except a case of some really trivial ones ).

Here we come with the macro-COSMOS design.

It is your responsibility to design a higher-space algorithm, how to make many isolated FSA-primitives converse and find an agreement in accord with the evolving many-to-many conversation. Yes. The library gives you "just" primitive building blocks (very powerful, out of doubt). But it is your responsibility to make the "outer-space" work for your needs.

And this can and typically is complex.

Well, if that would be trivial, then it most probably would have been already included "inside" the library, wouldn't it?

Where to go next?

Perhaps a best next step one may do is IMHO to make a step towards a bit more global view, which may and will sound complicated for the first few things one tries to code with ZeroMQ, but if you at least jump to the page 265 of Pieter Hintjens' book, Code Connected, Volume 1, if it were not the case of reading step-by-step there.

One can start to realise the way, how it is possible to start "programming" the macro-COSMOS of FSA-primitives, so as to form a higher-order-FSA-of-FSAs, that can and will solve all the ProblemDOMAIN specific issues.

First have an un-exposed view on the Fig.60Republishing Updates and Fig.62HA Clone Server pair and try only after that to go back to the roots, elements and details.