Diffusion 6.7 Release Notes

6.7.1 (12 August 2021)

6.7.0 (6 August 2021)

New features in 6.7.0

CDC Adapter

26853: CDC (Change Data Capture) adapter for databases

The CDC (Change Data Capture) adapter is available with this release. It can be used to connect a Diffusion server (or server cluster) to a database server. It uses Debezium under the hood to connect with databases. Currently, the adapter is supported for production use with MySQL and PostgreSQL databases. Connecting to other databases may work, but is not supported.

Client

20948: Missing topic notifications: providing more information about the origin

Previously the only information that a missing topic notification provided was the selector subscribed to (that matched no topics) and the session identifier of the subscriber. The notification has now been enhanced to provide all of the session properties of the originating session, as well as a list of the names of servers that the notification has passed through.

Server

23774: Missing topic notifications via remote server connections

From this release, missing topic notifications can be propagated via remote servers from secondary to primary servers. This is enabled by configuring the remote server definition with a new property indicating a selector that specifies which missing topic notifications should be propagated. By default, the behavior is as before (missing topic notifications not propagated), but propagation can be enabled by configuring the new 'missing topic notification filter' value on the remote server.

Topic Views

24508: New JSON patch clause

A new clause called 'patch' has been added to topic views. This allows a JSON patch to be applied to JSON values. Views containing this clause will only be applied to JSON topics.

26294: Topic type conversions

Topic views have a new 'type' clause which allows you to specify a topic type for the generated reference topic which may be different from the source topic type. If the selected source topic (or current transformed value) can be converted to the specified type, a reference topic of the specified type will be created; otherwise, there will be no reference topic. For a full specification of the supported conversions see the API documentation or the user manual.

Topics

22432: Session trees

A new 'session trees' feature is included in this release.

A session tree is a customized version of the topic tree presented to a session, adjusted by applying declarative branch mapping rules maintained by the server. Each session can be presented with a unique session tree based upon its session properties.

Session trees replace the previous 'routing topics' functionality but providing a much more powerful and flexible rules-based approach.

See the user manual and client API documentation for a full description of session trees.

Improvements in 6.7.0

C Client

24042: Close session using session ID

The C client can now close other sessions via the method diffusion_client_close_with_session which receives a sessionID in its parameters.

Client

22602: API support for creating and maintaining metric collectors

In previous releases, metric collectors could only be created and maintained using the Diffusion management console or JMX. There is now a new Metrics feature in the Client APIs that allows for creation and maintenance of metric collectors.

JavaScript Client

24858: Map non-string values to strings, for values of Options.properties

When setting session properties in diffusion.connect(), non-string values can be passed as property values.

Licensing

24000: Nightly license check expiry warning

Nightly license check now also warns about expiry.

Python Client

24430: Topic removal returns the number of topics removed

Topics.remove_topic now returns a TopicRemovalResult. The TopicRemovalResult.removed_count property returns the number of non-reference topics that were removed as a result of the operation.

26146: Support indefinite length CBOR encoding

This change implements indefinite encoding of CBOR, the preferred variant of the binary protocol used to communicate with the Diffusion server, for the Python module via an additional dependency called 'diffusion-cbor'.

To install from PyPi, proceed as before.

To install from the download package, do the following:

1. Unzip the Python client.
2. Unzip the Diffusion CBOR Python package corresponding to your platform (macOS, Windows or Linux) and run:
`pip install {path to python client wheel} --find-links {path to unpacked CBOR package)`

Other platforms are not yet supported.

27245: Local installation method for Python Client updated

Documentation now reflects the new local installation method.

Replication

27038: Avoid delta calculations when shipping large numbers of time series topic events to replica servers

In previous releases, if the cluster topology changed, replication of time series topics that have many events could cause the server to stall (with reports of busy multiplexer threads) due to the CPU cost of delta calculations. From this release, delta streams are no longer used when replicating large time series histories.

Deprecations in 6.7.0

.NET Client

27219: Callbacks deprecated

All methods that use callbacks have been deprecated at this release. Task based variants should be used in preference.

Java Client

26739: Callbacks are deprecated

All methods that use callbacks have been deprecated in this release. Equivalents that use CompletableFutures should be used instead.

26846: ClientControl.sessionIdFromString deprecated in favor of new Diffusion.sessionIdFromString


27204: ClientControl.setQueueEventHandler(QueueEventHandler) deprecated

The method ClientControl.setQueueEventHandler(QueueEventHandler) has been deprecated in favor of the newer setQueueEventHandler(QueueEventStream) method.

27205: ClientControl.setSessionPropertiesListener(SessionPropertiesListener) deprecated

The ClientControl.setSessionPropertiesListener(SessionPropertiesListener) method has been deprecated in favor of the newer setSessionPropertiesListener(SessionPropertiesStream) method.

27206: TopicControl.addMissingTopicHandler(String, MissingTopicHandler) deprecated

The TopicControl.addMissingTopicHandler(String, MissingTopicHandler) method has been deprecated in favor of the newer addMissingTopicHandler(String, MissingTopicNitificationStream) method.

27207: TopicControl.newSpecification deprecated

The TopicControl.newSpecification method has been deprecated in favor of the newer Diffusion.newTopicSpecification method.

Topics

26333: Routing topics are deprecated

Routing topics are deprecated at this release and will be removed in a future release. The more powerful SessionTrees feature introduced in this release should be used in their place.

Removals in 6.7.0

Client

23287: MessagingControl feature interface removed

The deprecated MessagingControl feature has been removed. All its functionality has been moved to the Messaging feature.

26331: Some unused TopicAddFailReason values have been removed

Unused TopicAddFailReason codes deprecated in Diffusion 6.2 or earlier have been removed from the client APIs.

Java Client

23283: com.pushtechnology.client.callbacks.TopicTreeHandler removed

The unused interface com.pushtechnology.client.callbacks.TopicTreeHandler has been removed.

Fixes in 6.7.0

.NET Client

27139: Credentials type is incorrect when authenticating

The .NET client was giving the wrong credentials type for custom credentials when authenticating. This has now been resolved.

Apple Client

26573: Apple client cannot connect securely to Diffusion Server via proxy

When the Apple client connected via secure WebSocket to a Diffusion server via a proxy, the connection would be downgraded to a non-secure WebSocket. This has now been fixed.

Client

27105: Applying JSON patch to a time series topic fails with server exception

The 'apply JSON patch' feature only works for JSON topics and should fail with an IncompatibleTopicException for any other type. However, if it was attempted for a time series topic with a JSON event type, a server side exception occurred and the correct exception was not reported to the client.

This has now been resolved. An attempt to apply a patch to a time series topic now results in the same exception being reported to the client as for other non-JSON topic types.

Console

26385: JSON data in time series history table still displays buffers

The Diffusion management console could previously display incorrect data in previews of past JSON time series topic values, when a 64 bit integer was present in the topic data. This has been fixed.

26450: Connection count on gateway tab can be misleading

The Diffusion management console could display confusing information about cluster servers when operating under a license with no connection limit. The console will now display "Unlimited" instead of "0" for the connection limit in this circumstance.

26607: Topic metric layout problem: wide tables overflow without providing usable scrollbar

The Diffusion management console tables for session and topic metrics could overflow the page horizontally on smaller displays. The tables are now horizontally scrollable within the page.

26652: Session properties do not appear

In Diffusion Management console releases prior to 6.6, the session browser view presented selected session properties when a particular session was selected. Due to a technical oversight, this feature was missing in 6.6. The feature has been restored, and all session properties for a particular session are now visible by selecting the session in the session browser view.

26653: Help text for $Principal and $Roles session properties is reversed

In the Diffusion management console, popover help text for the $Principal and $Roles session properties was shown for the wrong property. This has been corrected.

26983: Console presents inconsistent time zones in time series inspector

In the Diffusion management console, the time series inspector accepted time ranges in the local timezone, but displayed the timestamps of results in the UTC timezone. Event timestamps are now displayed in the local timezone, along with their offset from UTC.

Demos

22654: Racing Cars demo hangs when started without Diffusion server

Racing car demo now displays meaningful error when unable to connect to a Diffusion server.

Java & Android Client

26318: Header parsing during proxy authentication is not case insensitive

An issue where headers such as Keep-Alive and Connection were not checked ignoring case during proxy authentication has been fixed.

JavaScript Client

26226: JSON deltas fail with indefinite length arrays and objects

JSON deltas failed when the JSON was encoded using CBOR with indefinite length arrays and objects. Now JSON deltas work with definite and indefinite length encoded CBOR.

26439: Deregistration callbacks return Promises but this is not reflected in the TypeScript API or documentation

Deregistration callbacks in MissingTopicHandler and SessionPropertiesListener return a Promise that resolves when the listener has been deregistered. This is now reflected in the documentation and in the TypeScript API.

26441: DataType.writeValue can accept null but this is not reflected in the TypeScript API

writeValue for primitive and JSON datatypes can accept null as a value. This is now reflected in the TypeScript API

26662: int64 docs reference overflowing number

The documentation of int64 referenced a number that would cause an overflow. It has now been replaced with a number that fits into a signed 64-bit integer.

26799: JSON.jsonDiff(original) does not take arbitrary objects as documented

Previously, JSON.jsonDiff(original) did not take arbitrary plain objects as documented. This has been fixed.

26860: Subscribing and appending to time series topics fails with shared sessions

There was a bug preventing plain data value streams to subscribe to time series topics when using shared sessions. This has been resolved.

26981: Error when trying to fetch time series topics using AnyDataType

FetchRequest threw an error when trying to obtain values from time series topics using AnyDataType. Now it will return the time series event on the FetchResult.

27136: Topic properties documentation issues

The documentation of topic properties in TopicSpecification contained incorrect entries and has been fixed.

MQTT

26694: Improve error handling of MQTT payloads that can't be mapped to Diffusion topic values

In previous releases, if an MQTT session attempted to publish a payload that could not be converted to an appropriate topic value, the server would log a stack trace and terminate the session. For example, publishing a string that could not be converted to an integer to an INT64 topic, or publishing a non-JSON string to a JSON topic.

From this release, the server handles these scenarios more gracefully, logging a warning message, and rejecting the publication with the MQTT error code 0x83 (implementation-specific error).

Python Client

26980: mypy objects to missing mock type hints in Python 3.7/3.8 on some platforms

This change addresses how type checking in the build process handles missing type stubs. It now pulls in any missing type stubs automatically where available, and ignores the remainder. There were no changes in shipping code in the latest version.

Replication

26370: PUSH-000753 not logged when quorum met

Due to a bug introduced in Diffusion 6.6, the server did not log a message when quorum was met. This been rectified in this release.

26407: Cluster routing could fail to find a handler hosted by a peer

When a server is part of a cluster, if it has no local handler for various types of event such as messages, missing topic notifications, and authentication requests, but a peer server does, the event will be forwarded to the peer for processing.

Due to a bug in previous releases, some requests would not be forwarded even though a peer with a suitable handler existed. The bug has been fixed in this release.

Security

26950: TOPIC_OWNER topic property not respected when a session changes its principal

When a session re-authenticates using the "change principal" API, the server re-evaluates its subscriptions. In previous releases, the re-evaluation failed to take account of the TOPIC_OWNER topic property. This bug has been fixed in this release. Now a session's subscriptions can change if it re-authenticates using a different principal, so that it gains or loses ownership of a topic.

Server

27063: Remote server reconnection has memory leak

Repeated reconnections of remote servers could lead to a memory leak. This has now been resolved.

27096: Thread deadlock between DefaultPicoContainer and RemoteServerLinkManagerImpl

When using remote topic views, a thread deadlock could occur. This would manifest in a thread dump as a deadlock between DefaultPicoContainer and RemoteServerLinkManagerImpl. This has now been resolved.

System Monitoring/Statistics

26581: JMX service binding to all interfaces, despite configuration

Diffusion 6.5.0 added support for JMX over TLS, and made this the default configuration.

Due to a bug, when configured for TLS, the JMX service always bound to all local network interfaces regardless of whether a specific interface was specified by the host parameter in Management.xml. The bug has been fixed in this release, and the host parameter is now respected.

Topic Views

26584: Range query fails on time series reference topic created from a remote topic view

A query against a time series reference topic created from a remote topic view would fail, indicating that the topic was not a time series topic. This problem has now been resolved.

27134: NullPointerException evaluating a Time Series JSON topic with a view that contains an insert clause

A topic view could incorrectly select a time series JSON topic and result in a NullPointerException at the server if the view contained both a 'scalar' or 'as value' clause and a clause that applies only to JSON topics (such as insert or expand).
This has now been resolved.

Topics

26973: Time series range query returns incorrect values

In some situations, a time series topic range query could return incorrect values. This could be seen in the console or in range queries issued by clients. This has been resolved in this release.