Diffusion 6.7 Release Notes

6.7.2 (28 October 2021)

Fixes in 6.7.2

.NET Client

27629: Disconnection due to "Http fragmentation and extension not supported"

An issue was identified with the .Net client's handling of partial reads. This has now been addressed.

C Client

27408: Segmentation fault in add_time_series_stream

Fixed a segmentation fault that occurred when calling add_time_series_stream with a NULL session.

27554: hash_num_new is using minimum slots instead of maximum slots provided as parameter

hash_num_new now correctly uses the maximum number of slots when creating a hash map.

Console

27318: Unable to set remote server missing topic notification filter through console

The Diffusion management console did not allow the configuration of a topic notification filter while creating remote servers, functionality which was added to the server in Diffusion 6.7.0. This setting can now be configured through the console user interface.

27378: Topic view editor discards patch when attempting to edit existing topic view

When using the Diffusion management console to view existing topic views with a JSON patch clause, the console could incorrectly display the topic view without the JSON patch clause. This has been fixed.

27381: Topic paths with trailing spaces not handled correctly

In the Diffusion management console, there was no provision made for distinguishing topics whose paths differed in trailing whitespace. Behaviour in such cases has been improved.

Java & Android Client

27707: Java examples do not build out of the box

It was not possible to build the Java examples with "mvn package" without first adding a dependency for jackson-annotations. This has now been resolved.

Java Client

27754: Memory leak in Java client on multiple reconnections

Repeated reconnections from the Java client could cause a memory leak of session related objects in the client VM. This has now been resolved.

JavaScript Client

27569: TypeScript definition for RemoteServerBuilder.missingTopicNotificationFilter doesn't allow null parameter

The documentation of RemoteServerBuilder.missingTopicNotificationFilter states that a null parameter can be used to clear the filter. The TypeScript definition didn't allow null to be passed. Now, the type definition has been updated to allow a null parameter.

27697: Sessions can reconnect even if explicitly closed by another session

An issue has been resolved where the server allowed clients to reconnect during the reconnection timeout, even if they had been explicitly closed by another session. This would only occur if session replication was enabled.

Kafka Adapter

27259: Time series topic creation does not work in Kafka adapter

Fixed a bug where creating timeseries Diffusion topic was not working when publishing to DIffusion from Kafka Topics.

27261: Editing Diffusion topic related detail in Kafka adapter from console does not work

Fixed a bug where updating Diffusion publisher service configuration during runtime prevented updates to be published to updated Diffusion topics.

Python Client

27352: Recursively decode Model-based objects

Fixes an issue where some pydantic.BaseModel-based objects were not being fully decoded from the CBOR. This only affected Session Metrics.

Replication

27323: "replicated-topics-restored" start condition does not work

Connectors can be configured not to accept connections until a set of conditions is satisfied.

Due to a coding error in previous releases, the "replicated-topics-restored" condition was never triggered. This has been corrected in this release. The condition is satisfied after a server has joined the cluster and received all of the topic data from existing members of the cluster. The server will log a PUSH-000834 INFO message when this happens.

27333: Correct cluster recovery of time-series topics

In previous releases, due to a coding error, re-distributing time series topic data when servers join and leave a cluster did not scale to large numbers of time series events. This could cause protracted instability whenever the cluster topology changed. The problem has been fixed in this release.

Server

27720: Possible leak of sessions (and session metrics) that time out during connection

In certain situations, a client session failing to connect due to a timeout could lead to a memory leak where the server side client object remains. This would also affect metrics as the failed session would remain in the 'open' and 'connected' counts.

This problem has now been resolved.

6.7.1 (12 August 2021)

6.7.0 (6 August 2021)

New features in 6.7.0

CDC Adapter

26853: CDC (Change Data Capture) adapter for databases

The CDC (Change Data Capture) adapter is available with this release. It can be used to connect a Diffusion server (or server cluster) to a database server. It uses Debezium under the hood to connect with databases. Currently, the adapter is supported for production use with MySQL and PostgreSQL databases. Connecting to other databases may work, but is not supported.

Client

20948: Missing topic notifications: providing more information about the origin

Previously the only information that a missing topic notification provided was the selector subscribed to (that matched no topics) and the session identifier of the subscriber. The notification has now been enhanced to provide all of the session properties of the originating session, as well as a list of the names of servers that the notification has passed through.

Server

23774: Missing topic notifications via remote server connections

From this release, missing topic notifications can be propagated via remote servers from secondary to primary servers. This is enabled by configuring the remote server definition with a new property indicating a selector that specifies which missing topic notifications should be propagated. By default, the behavior is as before (missing topic notifications not propagated), but propagation can be enabled by configuring the new 'missing topic notification filter' value on the remote server.

Topic Views

24508: New JSON patch clause

A new clause called 'patch' has been added to topic views. This allows a JSON patch to be applied to JSON values. Views containing this clause will only be applied to JSON topics.

26294: Topic type conversions

Topic views have a new 'type' clause which allows you to specify a topic type for the generated reference topic which may be different from the source topic type. If the selected source topic (or current transformed value) can be converted to the specified type, a reference topic of the specified type will be created; otherwise, there will be no reference topic. For a full specification of the supported conversions see the API documentation or the user manual.

Topics

22432: Session trees

A new 'session trees' feature is included in this release.

A session tree is a customized version of the topic tree presented to a session, adjusted by applying declarative branch mapping rules maintained by the server. Each session can be presented with a unique session tree based upon its session properties.

Session trees replace the previous 'routing topics' functionality but providing a much more powerful and flexible rules-based approach.

See the user manual and client API documentation for a full description of session trees.

Improvements in 6.7.0

C Client

24042: Close session using session ID

The C client can now close other sessions via the method diffusion_client_close_with_session which receives a sessionID in its parameters.

Client

22602: API support for creating and maintaining metric collectors

In previous releases, metric collectors could only be created and maintained using the Diffusion management console or JMX. There is now a new Metrics feature in the Client APIs that allows for creation and maintenance of metric collectors.

JavaScript Client

24858: Map non-string values to strings, for values of Options.properties

When setting session properties in diffusion.connect(), non-string values can be passed as property values.

Licensing

24000: Nightly license check expiry warning

Nightly license check now also warns about expiry.

Python Client

24430: Topic removal returns the number of topics removed

Topics.remove_topic now returns a TopicRemovalResult. The TopicRemovalResult.removed_count property returns the number of non-reference topics that were removed as a result of the operation.

26146: Support indefinite length CBOR encoding

This change implements indefinite encoding of CBOR, the preferred variant of the binary protocol used to communicate with the Diffusion server, for the Python module via an additional dependency called 'diffusion-cbor'.

To install from PyPi, proceed as before.

To install from the download package, do the following:

1. Unzip the Python client.
2. Unzip the Diffusion CBOR Python package corresponding to your platform (macOS, Windows or Linux) and run:
`pip install {path to python client wheel} --find-links {path to unpacked CBOR package)`

Other platforms are not yet supported.

27245: Local installation method for Python Client updated

Documentation now reflects the new local installation method.

Replication

27038: Avoid delta calculations when shipping large numbers of time series topic events to replica servers

In previous releases, if the cluster topology changed, replication of time series topics that have many events could cause the server to stall (with reports of busy multiplexer threads) due to the CPU cost of delta calculations. From this release, delta streams are no longer used when replicating large time series histories.

Deprecations in 6.7.0

.NET Client

27219: Callbacks deprecated

All methods that use callbacks have been deprecated at this release. Task based variants should be used in preference.

Java Client

26739: Callbacks are deprecated

All methods that use callbacks have been deprecated in this release. Equivalents that use CompletableFutures should be used instead.

26846: ClientControl.sessionIdFromString deprecated in favor of new Diffusion.sessionIdFromString


27204: ClientControl.setQueueEventHandler(QueueEventHandler) deprecated

The method ClientControl.setQueueEventHandler(QueueEventHandler) has been deprecated in favor of the newer setQueueEventHandler(QueueEventStream) method.

27205: ClientControl.setSessionPropertiesListener(SessionPropertiesListener) deprecated

The ClientControl.setSessionPropertiesListener(SessionPropertiesListener) method has been deprecated in favor of the newer setSessionPropertiesListener(SessionPropertiesStream) method.

27206: TopicControl.addMissingTopicHandler(String, MissingTopicHandler) deprecated

The TopicControl.addMissingTopicHandler(String, MissingTopicHandler) method has been deprecated in favor of the newer addMissingTopicHandler(String, MissingTopicNitificationStream) method.

27207: TopicControl.newSpecification deprecated

The TopicControl.newSpecification method has been deprecated in favor of the newer Diffusion.newTopicSpecification method.

Topics

26333: Routing topics are deprecated

Routing topics are deprecated at this release and will be removed in a future release. The more powerful SessionTrees feature introduced in this release should be used in their place.

Removals in 6.7.0

Client

23287: MessagingControl feature interface removed

The deprecated MessagingControl feature has been removed. All its functionality has been moved to the Messaging feature.

26331: Some unused TopicAddFailReason values have been removed

Unused TopicAddFailReason codes deprecated in Diffusion 6.2 or earlier have been removed from the client APIs.

Java Client

23283: com.pushtechnology.client.callbacks.TopicTreeHandler removed

The unused interface com.pushtechnology.client.callbacks.TopicTreeHandler has been removed.

Fixes in 6.7.0

.NET Client

27139: Credentials type is incorrect when authenticating

The .NET client was giving the wrong credentials type for custom credentials when authenticating. This has now been resolved.

Apple Client

26573: Apple client cannot connect securely to Diffusion Server via proxy

When the Apple client connected via secure WebSocket to a Diffusion server via a proxy, the connection would be downgraded to a non-secure WebSocket. This has now been fixed.

Client

27105: Applying JSON patch to a time series topic fails with server exception

The 'apply JSON patch' feature only works for JSON topics and should fail with an IncompatibleTopicException for any other type. However, if it was attempted for a time series topic with a JSON event type, a server side exception occurred and the correct exception was not reported to the client.

This has now been resolved. An attempt to apply a patch to a time series topic now results in the same exception being reported to the client as for other non-JSON topic types.

Console

26385: JSON data in time series history table still displays buffers

The Diffusion management console could previously display incorrect data in previews of past JSON time series topic values, when a 64 bit integer was present in the topic data. This has been fixed.

26450: Connection count on gateway tab can be misleading

The Diffusion management console could display confusing information about cluster servers when operating under a license with no connection limit. The console will now display "Unlimited" instead of "0" for the connection limit in this circumstance.

26607: Topic metric layout problem: wide tables overflow without providing usable scrollbar

The Diffusion management console tables for session and topic metrics could overflow the page horizontally on smaller displays. The tables are now horizontally scrollable within the page.

26652: Session properties do not appear

In Diffusion Management console releases prior to 6.6, the session browser view presented selected session properties when a particular session was selected. Due to a technical oversight, this feature was missing in 6.6. The feature has been restored, and all session properties for a particular session are now visible by selecting the session in the session browser view.

26653: Help text for $Principal and $Roles session properties is reversed

In the Diffusion management console, popover help text for the $Principal and $Roles session properties was shown for the wrong property. This has been corrected.

26983: Console presents inconsistent time zones in time series inspector

In the Diffusion management console, the time series inspector accepted time ranges in the local timezone, but displayed the timestamps of results in the UTC timezone. Event timestamps are now displayed in the local timezone, along with their offset from UTC.

Demos

22654: Racing Cars demo hangs when started without Diffusion server

Racing car demo now displays meaningful error when unable to connect to a Diffusion server.

Federation

27063: Remote server reconnection has memory leak

Repeated reconnections of remote servers could lead to a memory leak. This has now been resolved.

Java & Android Client

26318: Header parsing during proxy authentication is not case insensitive

An issue where headers such as Keep-Alive and Connection were not checked ignoring case during proxy authentication has been fixed.

JavaScript Client

26226: JSON deltas fail with indefinite length arrays and objects

JSON deltas failed when the JSON was encoded using CBOR with indefinite length arrays and objects. Now JSON deltas work with definite and indefinite length encoded CBOR.

26439: Deregistration callbacks return Promises but this is not reflected in the TypeScript API or documentation

Deregistration callbacks in MissingTopicHandler and SessionPropertiesListener return a Promise that resolves when the listener has been deregistered. This is now reflected in the documentation and in the TypeScript API.

26441: DataType.writeValue can accept null but this is not reflected in the TypeScript API

writeValue for primitive and JSON datatypes can accept null as a value. This is now reflected in the TypeScript API

26662: int64 docs reference overflowing number

The documentation of int64 referenced a number that would cause an overflow. It has now been replaced with a number that fits into a signed 64-bit integer.

26799: JSON.jsonDiff(original) does not take arbitrary objects as documented

Previously, JSON.jsonDiff(original) did not take arbitrary plain objects as documented. This has been fixed.

26860: Subscribing and appending to time series topics fails with shared sessions

There was a bug preventing plain data value streams to subscribe to time series topics when using shared sessions. This has been resolved.

26981: Error when trying to fetch time series topics using AnyDataType

FetchRequest threw an error when trying to obtain values from time series topics using AnyDataType. Now it will return the time series event on the FetchResult.

27136: Topic properties documentation issues

The documentation of topic properties in TopicSpecification contained incorrect entries and has been fixed.

MQTT

26694: Improve error handling of MQTT payloads that can't be mapped to Diffusion topic values

In previous releases, if an MQTT session attempted to publish a payload that could not be converted to an appropriate topic value, the server would log a stack trace and terminate the session. For example, publishing a string that could not be converted to an integer to an INT64 topic, or publishing a non-JSON string to a JSON topic.

From this release, the server handles these scenarios more gracefully, logging a warning message, and rejecting the publication with the MQTT error code 0x83 (implementation-specific error).

Python Client

26980: mypy objects to missing mock type hints in Python 3.7/3.8 on some platforms

This change addresses how type checking in the build process handles missing type stubs. It now pulls in any missing type stubs automatically where available, and ignores the remainder. There were no changes in shipping code in the latest version.

Replication

26370: PUSH-000753 not logged when quorum met

Due to a bug introduced in Diffusion 6.6, the server did not log a message when quorum was met. This been rectified in this release.

26407: Cluster routing could fail to find a handler hosted by a peer

When a server is part of a cluster, if it has no local handler for various types of event such as messages, missing topic notifications, and authentication requests, but a peer server does, the event will be forwarded to the peer for processing.

Due to a bug in previous releases, some requests would not be forwarded even though a peer with a suitable handler existed. The bug has been fixed in this release.

Security

26950: TOPIC_OWNER topic property not respected when a session changes its principal

When a session re-authenticates using the "change principal" API, the server re-evaluates its subscriptions. In previous releases, the re-evaluation failed to take account of the TOPIC_OWNER topic property. This bug has been fixed in this release. Now a session's subscriptions can change if it re-authenticates using a different principal, so that it gains or loses ownership of a topic.

Server

27096: Thread deadlock between DefaultPicoContainer and RemoteServerLinkManagerImpl

When using remote topic views, a thread deadlock could occur. This would manifest in a thread dump as a deadlock between DefaultPicoContainer and RemoteServerLinkManagerImpl. This has now been resolved.

System Monitoring/Statistics

26581: JMX service binding to all interfaces, despite configuration

Diffusion 6.5.0 added support for JMX over TLS, and made this the default configuration.

Due to a bug, when configured for TLS, the JMX service always bound to all local network interfaces regardless of whether a specific interface was specified by the host parameter in Management.xml. The bug has been fixed in this release, and the host parameter is now respected.

Topic Views

26584: Range query fails on time series reference topic created from a remote topic view

A query against a time series reference topic created from a remote topic view would fail, indicating that the topic was not a time series topic. This problem has now been resolved.

27134: NullPointerException evaluating a Time Series JSON topic with a view that contains an insert clause

A topic view could incorrectly select a time series JSON topic and result in a NullPointerException at the server if the view contained both a 'scalar' or 'as value' clause and a clause that applies only to JSON topics (such as insert or expand).
This has now been resolved.

Topics

26973: Time series range query returns incorrect values

In some situations, a time series topic range query could return incorrect values. This could be seen in the console or in range queries issued by clients. This has been resolved in this release.

Known Issues

Topic Views

27594: Reference topics retained by 'preserve topics' are not persisted across server instances or cluster

A new 'preserve topics' clause was introduced to topic views in release 6.6. This clause means that reference topics created by a view (that have a path dependent upon the source topic value) are retained until the source topic is removed or the topic view is removed. Though this is true in the context of a single server instance, it is not the case if the server is restarted as all such topics created during the previous server instance will be lost. It is also not the case if a new server enters a cluster as the new server will only have reference topics generated from the point in time where it joined the cluster and will not reflect reference topics previously created within other cluster peers.

This issue occurs because reference topics are not persisted, either to file or across the cluster.

This issue will be resolved in a future release.