Using conflation
You can configure how and when conflation is applied. Configuration works differently for legacy topic types (stateless, single value and record).
Conflation policies
Conflation policies control how conflation is applied to a topic. You can set conflation policy for a topic with a topic property.
- off
- conflate (default)
- unsubscribe
- always
"off" disables all conflation for the topic. Topic updates will never be merged or discarded.
"conflate" automatically conflates topic updates when back pressure is detected by the server (that is, when the outgoing message queue exceeds the maximum allowed size in bytes or number of messages).
"unsubscribe" automatically unsubscribes the topic when back pressure is detected by the server, with a BACK_PRESSURE message. The unsubscription is not persisted to the cluster, so if a session fails over to a different server it will be resubscribed to the topic. This policy is useful for topics that are not essential to the application, and can be discarded in back pressure situations without affecting the main function of the application.
"always" automatically conflates topic updates as they are queued for the session. This policy ensures only the latest update is queued for the topic, minimising the server memory and network bandwidth used by the session, but potentially increasing the processor cost of conflation.
If no policy is set, the "conflate" policy is applied.
Conflation process
The conflation process considers the value and delta updates in the queue, and the current topic value (unless DONT_RETAIN_LAST_VALUE is enabled). It reduces the queued updates to a single value or a composite delta, whichever requires the fewest bytes to send.
Under the default "conflate" policy, no conflation is applied until there is a new message to send to a session with a full queue. The whole queue is then conflated, topic by topic. If conflation is not enough to bring the queue size under the configured limit, the server will close the session.
Legacy conflation policies
- How messages are matched
- Whether replacement is done in place or by appending
- How to merge the two messages
Legacy conflation policies are configured within Server.xml.
If an "off" or "unsubscribe" policy is set for one of the legacy topic types (stateless, single value or record) using the topic properties system described above, that policy will be applied and override the settings in Server.xml.
If an "always" or "conflate" policy is set, the Server.xml configuration will be applied.
Legacy conflation process
When legacy conflation is enabled for a client, every time a new update is enqueued for the client it is considered for conflation.
- The Diffusion™ server checks whether a conflation policy is mapped to the topic that the update is published on.
- If a policy is mapped to the topic, the matching criteria of the policy is
used to scan the updates for that topic that exist on the queue (from newest
to oldest) for an existing update that matches the one being queued. If no
match is found, the new update is queued at the end of the queue as normal.
Note:
Fast caches and lookup mechanisms are used to find policies and updates in the queue. The whole client queue is not scanned when looking for a match, only updates for the same topic.
If default matching (by topic) is used, no comparison with the existing updates is required. This means that the conflation mechanism is as efficient as possible.
- If the matching criteria finds a matching update in the queue, the
conflation policy defines the following behaviors:
- Whether the update to queue as a result of conflation is the new update or an update produced by merging the content of the matching update and new update.
- Whether the update to queue replaces the matching update or the matching update is removed and the new update added to the end of the queue.
Legacy conflation occurs on a client-by-client basis in the multiplexer thread. Results of merge actions are cached to ensure that merges of identical message pairs are not repeated for different clients.
This page last modified: 2018/04/19