Fan-out
Consider whether to use fan-out to replicate topic information from primary servers on one or more secondary servers.
Why use fan-out?
Having a primary server feed out updates to a number of secondary servers provides a solution where the same topics and data are available from multiple servers. You can use this solution to load balance a large number of client connections across a set of Diffusion™ servers and provide those clients with the same access to data.
How fan-out works
Fan-out is configured on the secondary server or secondary servers in the solution.
- A secondary server connects to a primary server as a client.
- The secondary server subscribes to a set of topics on the primary server.
This set of topics is defined by a selector in the configuration of the secondary server.
- The secondary server replicates the subscribed topics locally.
- When updates are made to the topics on the primary server, the secondary server receives these updates through the standard pub-sub feature in the same way as any other client of the primary server.
- The secondary server applies the updates to its replica topics.
- Any clients subscribed to a replica topic on the secondary server receive the updates through the standard pub-sub feature.
- If a topic is removed at the primary server, the secondary server removes its replica topic.
- If a topic is added at the primary server that matches the set of topics subscribed by the secondary server, the secondary server creates a local replica topic.
A secondary server can connect as a client and subscribe to topics on more than one primary server. However, ensure that the secondary server does not attempt to replicate the same topic from multiple sources as this can cause the data on the topic to be incorrect.
Creating topics on the primary server is an asynchronous action, because of this a client or publisher that creates a topic on the primary server receives a completed callback saying that the topic has been created. However, receiving this callback does not indicate that the topic has been replicated by fan-out and created on a secondary server.
Topic types supported by fan-out
- JSON
- Binary
- Single value topics
- Record topics
- Stateless topics
- Slave topics
The order of topic creation on the secondary server can prevent slave topics from being replicated. For example, if a slave topic refers to a topic that does not yet exist because it is in a branch not yet replicated or because it is lower down the link hierarchy.
- Routing topics
To use fan-out with routing topics, the routing subscription handlers for a routing topic must be registered at the primary and all secondary servers. The routing logic provided by the handlers on the primary and secondary server must be identical.
- DEPRECATED: Paged string topics
- DPERECATED: Paged record topics
- DEPRECATED: Custom topics
The custom class for that topic must be available on the classpath of all Diffusion servers replicating that topic.
- DEPRECATED: Protocol buffer topics
The compiled .proto file that defines the format of the data on a protocol buffer topic must be available on the classpath of all Diffusion servers replicating that topic.
Topic replication and fan-out
A secondary server cannot replicate the same topic from more than one primary server or multiple times from the same primary server. Validation of the root path part of the selectors is in place to prevent this occurring, but the use of regular expressions in topic selectors can result in an overlap of replication which can cause problems.
Missing topic notifications generated by subscription or fetch requests to a secondary server are propagated to missing topic handlers registered against the primary servers. For more information, see Using missing topic notifications with fan-out.
Fan-out and load balancers
If you add a secondary server to a load balancer pool before all topics have propagated from the primary server, it can result in a large number of messages being generated, leading to MESSAGE_QUEUE_LIMIT_REACHED errors appearing in the logs.
If you experience this problem, introduce a delay between enabling fan-out and adding any of the secondary servers to a load balancer pool. There is currently no built-in way to determine when propagation is complete, so you will need to experiment to find out how long the delay needs to be for your configuration.
Reconnection and disconnection
You can configure fan-out servers to use the standard reconnect mechanism. If the connection between the secondary server and the primary server is lost, the secondary server can reconnect to the same session. However, if messages are lost between the primary and secondary server, the reconnection is aborted and the session closed. The secondary server must connect again to the primary server with a new session.
If a disconnection between the primary and secondary server results in the session being closed, the secondary server removes all the topics that it has replicated from that primary server. (Only topics explicitly defined by a selector are removed.) Clients subscribing to these topics on the secondary server become unsubscribed. If the secondary server connects again to that primary server with a new session, the secondary server recreates the topics. Clients connecting to the secondary server become resubscribed to the topics.
This page last modified: 2017/06/14