Fan-out | Documentation

A fan-out distribution comprises many servers that host the same topic or topics. When the topic is updated on a primary server, the update is fanned out to replica topics on secondary servers.

Note: Remote topic views are generally a superior alternative to fan-out. They are easier to configure and can also be combined with other topic view capabilities to transform data. Fan-out, however can be used if all that is required is simple one to one mapping of topics from primary to secondary servers as it has a smaller memory footprint.

How fan-out works

Fan-out is configured on the secondary server or secondary servers in the solution.

A fan-out solution comprising one primary server and two secondary servers. The secondary servers have fan-out configured for branches of the topic tree on the primary server.

A secondary server connects to a primary server as a client.
The secondary server subscribes to a set of topics on the primary server.
This set of topics is defined by a selector in the configuration of the secondary server.
The secondary server replicates the subscribed topics locally.
When updates are made to the topics on the primary server, the secondary server receives these updates through the standard pub-sub feature in the same way as any other client of the primary server.
The secondary server applies the updates to its replica topics.
Any clients subscribed to a replica topic on the secondary server receive the updates through the standard pub-sub feature.
If a topic is removed at the primary server, the secondary server removes its replica topic.
If a topic is added at the primary server that matches the set of topics subscribed by the secondary server, the secondary server creates a local replica topic.

A secondary server can connect as a client and subscribe to topics on more than one primary server. However, ensure that the secondary server does not attempt to replicate the same topic from multiple sources as this can cause the data on the topic to be incorrect.

Whether a server is "primary" or "secondary" is a property of a particular fan-out link. You can create a configuration where a server is primary for one fan-out link, but secondary for another.

Creating topics on the primary server is an asynchronous action, because of this a client that creates a topic on the primary server receives a completed callback saying that the topic has been created. However, receiving this callback does not indicate that the topic has been replicated by fan-out and created on a secondary server.

Topic aliasing is not supported for topics that are replicated by fan-out. Ensure that aliasing is not enabled at the primary server.

Topic replication and fan-out

A secondary server cannot replicate the same topic from more than one primary server or multiple times from the same primary server. Validation of the path prefix of the selectors is in place to prevent this occurring, but the use of regular expressions in topic selectors can result in an overlap of replication which can cause problems.

Missing topic notifications generated by subscription or fetch requests to a secondary server are propagated to missing topic handlers registered against the primary servers. For more information, see Using missing topic notifications with fan-out.

Fan-out readiness

If you add a secondary server to a load balancer pool before all topics have propagated from the primary server, it can result in a large number of messages being generated, leading to MESSAGE_QUEUE_LIMIT_REACHED errors appearing in the logs.

As of Diffusion™ 6.4, you can avoid this problem using the fan-out readiness start condition configured in etc/Connectors.xml. If you enable fan-out readiness, the connector will not be started until a fanout link with specified connection and link names exists, has connected to the primary server, and has synchronized the topics from the primary server that match its configuration.

Reconnection and disconnection

You can configure fan-out servers to use the standard reconnect mechanism. If the connection between the secondary server and the primary server is lost, the secondary server can reconnect to the same session. However, if messages are lost between the primary and secondary server, the reconnection is aborted and the session closed. The secondary server must connect again to the primary server with a new session.

If a disconnection between the primary and secondary server results in the session being closed, the secondary server removes all the topics that it has replicated from that primary server. (Only topics explicitly defined by a selector are removed.) Clients subscribing to these topics on the secondary server become unsubscribed. If the secondary server connects again to that primary server with a new session, the secondary server recreates the topics. Clients connecting to the secondary server become resubscribed to the topics.