Diffusion 6.7 Release Notes

6.7.7 (7 April 2022)

Fixes in 6.7.7

C Client

28453: OpenSSL not included in C Client

OpenSSL is now internally linked in the Diffusion C Client.
Existing applications no longer need to link ssl to compile an executable.

Replication

28297: Excessive memory footprint in cluster partition log compaction

The log compaction process has now been tuned, significantly reducing the working memory required to handle a series of large messages sent to a replicated topic.

Server

28540: Failure to update to a topic can be logged twice

In previous releases, a PUSH-000464 message could be logged twice for a topic update failure. The bug has been corrected in this release.

6.7.6 (8 March 2022)

Fixes in 6.7.6

Console

28369: Cross site scripting vulnerability in the console logs tab

A remote code execution vulnerability in the server logs tab of the Diffusion Management Console has been resolved at this release.

Replication

26496: A server recovering from a persistent file can corrupt a cluster's replicated topic data

Due to a bug in previous releases, a server joining a cluster and recovering topics from a persistent store could corrupt the topic data in the cluster.

This bug had several symptoms, including inconsistencies between the cluster members' topic trees, and internal failures to apply delta updates. (E.g. PUSH-000843 ... state REPLICA cannot have delta applied by REPLICA).

The bug has been fixed in this release.

28267: Servers in a cluster are unresponsive when loading a large persistence file

Starting replicated servers in a Diffusion cluster could slow down to such a degree that it would appear that they had completely stalled when restoring from very large persistence files.This was due to unnecessary delta calculations occurring when restoring the cluster.
This has now been resolved.

Server

28246: New subscription inadvertently removed existing subscriptions

Due to a bug in previous releases, the topic selectors maintained by the server for a session could be corrupted by subscription and unsubscription operations. Specifically, the problem could be triggered if a session subscribed to a topic selector with a descendant pattern qualifier ("/", or "//"), for example "?a//", then later redundantly subscribed to a topic selector that is a strict sub-selector of the first one, for example "a/b". The bug could cause another topic selector to be removed in ways that were hard to predict.

The bug has been fixed in this release.

Topics

28308: IllegalArgumentException in TopicTreeNodeImpl

A concurrency bug in previous releases could corrupt topics in the topic tree. One side-effect is that a subsequent attempt to add a topic could fail with an IllegalArgumentException. The bug has been fixed in this release.

6.7.5 (7 February 2022)

Fixes in 6.7.5

Federation

28049: Remote server connection failure to connect stalls multiplexer

A problem with thread locking could cause multiplexers to stall if secondary remote servers fail to connect. This problem has now been resolved.

28207: Inbound threads can be indefinitely blocked by Remote Server API calls

If a remote server connection was blocking for a long time due to other issues then other calls to the remote server feature (create, remove, check, get) could also block inbound threads indefinitely.
This has now been changed so that such calls will time out if unable to proceed due to locks being held by remote server connections.

Persistence

28160: Persistence restore failure due to file corruption restores no topics

There is the possibility for topic persistence files to become corrupt. The most likely cause of this is some resource issue (memory or disk space) at the time of writing which can lead to a truncated file.
Previously, when restarting a server with such corrupt files the restore would be abandoned, files would be moved to the recovery directory, and the server started with no topics restored.
This fix allows the server to proceed with topics restored so far if a file corruption is detected. The faulty files will still be copied to the recovery directory but the current state already read from files will be written back to the persistence directory as a compacted file.
An error will be logged if this occurs, but as file corruption typically occurs at the end of the persistence files then in most cases this will mean that all topic state at the point of failure, except for the very last write, will be restored successfully.
Files copied to recovery are for diagnostic purposes only and should be manually deleted to save space. However, before deletion, they may be sent to Push Technology support for analysis.

Replication

28140: Connecting sessions that time out due to Hazelcast blocking never complete and remain in memory and metrics

When running in a cluster connecting sessions could time out if there was never a response from Hazelcast during the connection phase. This led to the session remaining in an unconnected state in the server memory and still showing in session metrics. This problems has now been resolved so that if the Hazelcast interaction does not complete the session closes tidily.

Security

28066: Changes to the security configuration fail in a cluster if the security store file is read-only

A bug was introduced in Diffusion 6.6.0 and later releases which corrupted the propagation of security configuration changes across a cluster if the corresponding security store file (SystemAuthentication.store, Security.store) is read-only.

Changing the file permissions so the security store files can be read but not written is supported, and can be useful if a separate mechanism is used to seed security configuration after a cluster is cold-started.

The bug has been fixed in this release. Security store changes are again propagated correctly across the cluster, regardless of whether the security store file is read-only.

6.7.4 (16 December 2021)

Fixes in 6.7.4

Security

28010: Upgrade log4j2 to address CVE-2021-45046 security vulnerability

The log4j2 logging library used by Diffusion has been upgraded to version 2.16.0. This addresses a critical security bug [CVS-2021-45046] in log4j2. See https://logging.apache.org/log4j/2.x/security.html for details.

6.7.3 (13 December 2021)

Fixes in 6.7.3

Console

27917: Nonsense on license page for commercial license

The Diffusion management console's license page could show some contradictory text when deployed with a commercial license. This has been resolved.

27919: Console does not allow connection timeout to be specified

The Diffusion management console did not allow a connection timeout to be specified at login. This option has been added.

Java Client

27831: Additional memory leak in Java client on multiple reconnections

Case 27754 in release 6.7.2 addressed a memory leak that could occur in the Java client on multiple reconnections. Though that fix partially addressed the problem there remained a situation where a memory leak was still possible. This has now been resolved.

JavaScript Client

27926: Connection timeout not configurable

The JavaScript client was missing an option to specify the connection timeout on establishing a session. This has been rectified.

27949: Authenticator throws 'Cannot read properties of null'

When closing an authenticator, it would throw a "REGISTERED_HANDLER_EXCEPTION TypeError: Cannot read properties of null". This has been fixed.

Replication

27966: A server joining a stable cluster should not merge topics recovered from file persistence

When servers configured for topic replication first form a cluster, the replicated topics are initialised from the servers persistent files. Each server that forms the initial cluster is responsible for recovering a proportion of the topics. While the cluster continues to run, persistent files are written but not read again.

Due to a bug in previous releases, a server joining a stable cluster could add topics from its persistent files. This bug has been fixed in this release.

27987: Unnecessary assertion from compaction

Due to a bug in previous releases, if the server was run with assertions enabled (-ea), topic replication could fail due to an assertion error during compaction of the persistence log. The bug has been fixed in this release.

Security

27973: Concurrency issue could lead to a corrupt permissions index

In previous releases, a bug in the code that creates internal index of security permissions could leave the index in a corrupt form. The bug has been fixed in this release.

27978: Upgrade log4j2 to address CVE-2021-44228 security vulnerability

The log4j2 logging library used by Diffusion has been upgraded to version 2.15.0. This addresses a critical security bug [CVS-2021-44228] in log4j2. See https://logging.apache.org/log4j/2.x/security.html for details.

27989: Setting READ_TOPIC permissions update could fail with an IllegalArgumentException

Due to a bug in previous releases, applying particular combinations of path permission assignments using the security control API could fail with an IllegalArgumentException. The bug has been fixed in this release.

Topic Views

27886: Inserts before patch clauses can cause indeterminate results

It was possible that having an 'insert' clause in a topic view specification before a 'patch' clause could produce indeterminate results and in some situations even lead to orphaned reference topics.
The validation of topic views has now been changed to ensure that any 'insert' clauses happen after 'patch' clauses. A failure will occur when parsing a topic view specification if this is not the case.

6.7.2 (28 October 2021)

Fixes in 6.7.2

.NET Client

27629: Disconnection due to "Http fragmentation and extension not supported"

An issue was identified with the .Net client's handling of partial reads. This has now been addressed.

C Client

27408: Segmentation fault in add_time_series_stream

Fixed a segmentation fault that occurred when calling add_time_series_stream with a NULL session.

27554: hash_num_new is using minimum slots instead of maximum slots provided as parameter

hash_num_new now correctly uses the maximum number of slots when creating a hash map.

Console

27318: Unable to set remote server missing topic notification filter through console

The Diffusion management console did not allow the configuration of a topic notification filter while creating remote servers, functionality which was added to the server in Diffusion 6.7.0. This setting can now be configured through the console user interface.

27378: Topic view editor discards patch when attempting to edit existing topic view

When using the Diffusion management console to view existing topic views with a JSON patch clause, the console could incorrectly display the topic view without the JSON patch clause. This has been fixed.

27381: Topic paths with trailing spaces not handled correctly

In the Diffusion management console, there was no provision made for distinguishing topics whose paths differed in trailing whitespace. Behaviour in such cases has been improved.

Java & Android Client

27707: Java examples do not build out of the box

It was not possible to build the Java examples with "mvn package" without first adding a dependency for jackson-annotations. This has now been resolved.

Java Client

27754: Memory leak in Java client on multiple reconnections

Repeated reconnections from the Java client could cause a memory leak of session related objects in the client VM. This has now been resolved.

JavaScript Client

27569: TypeScript definition for RemoteServerBuilder.missingTopicNotificationFilter doesn't allow null parameter

The documentation of RemoteServerBuilder.missingTopicNotificationFilter states that a null parameter can be used to clear the filter. The TypeScript definition didn't allow null to be passed. Now, the type definition has been updated to allow a null parameter.

27697: Sessions can reconnect even if explicitly closed by another session

An issue has been resolved where the server allowed clients to reconnect during the reconnection timeout, even if they had been explicitly closed by another session. This would only occur if session replication was enabled.

Kafka Adapter

27259: Time series topic creation does not work in Kafka adapter

Fixed a bug where creating timeseries Diffusion topic was not working when publishing to DIffusion from Kafka Topics.

27261: Editing Diffusion topic related detail in Kafka adapter from console does not work

Fixed a bug where updating Diffusion publisher service configuration during runtime prevented updates to be published to updated Diffusion topics.

Python Client

27352: Recursively decode Model-based objects

Fixes an issue where some pydantic.BaseModel-based objects were not being fully decoded from the CBOR. This only affected Session Metrics.

Replication

27323: "replicated-topics-restored" start condition does not work

Connectors can be configured not to accept connections until a set of conditions is satisfied.

Due to a coding error in previous releases, the "replicated-topics-restored" condition was never triggered. This has been corrected in this release. The condition is satisfied after a server has joined the cluster and received all of the topic data from existing members of the cluster. The server will log a PUSH-000834 INFO message when this happens.

27333: Correct cluster recovery of time-series topics

In previous releases, due to a coding error, re-distributing time series topic data when servers join and leave a cluster did not scale to large numbers of time series events. This could cause protracted instability whenever the cluster topology changed. The problem has been fixed in this release.

Server

27720: Possible leak of sessions (and session metrics) that time out during connection

In certain situations, a client session failing to connect due to a timeout could lead to a memory leak where the server side client object remains. This would also affect metrics as the failed session would remain in the 'open' and 'connected' counts.

This problem has now been resolved.

6.7.1 (12 August 2021)

6.7.0 (6 August 2021)

New features in 6.7.0

CDC Adapter

26853: CDC (Change Data Capture) adapter for databases

The CDC (Change Data Capture) adapter is available with this release. It can be used to connect a Diffusion server (or server cluster) to a database server. It uses Debezium under the hood to connect with databases. Currently, the adapter is supported for production use with MySQL and PostgreSQL databases. Connecting to other databases may work, but is not supported.

Client

20948: Missing topic notifications: providing more information about the origin

Previously the only information that a missing topic notification provided was the selector subscribed to (that matched no topics) and the session identifier of the subscriber. The notification has now been enhanced to provide all of the session properties of the originating session, as well as a list of the names of servers that the notification has passed through.

Server

23774: Missing topic notifications via remote server connections

From this release, missing topic notifications can be propagated via remote servers from secondary to primary servers. This is enabled by configuring the remote server definition with a new property indicating a selector that specifies which missing topic notifications should be propagated. By default, the behavior is as before (missing topic notifications not propagated), but propagation can be enabled by configuring the new 'missing topic notification filter' value on the remote server.

Topic Views

24508: New JSON patch clause

A new clause called 'patch' has been added to topic views. This allows a JSON patch to be applied to JSON values. Views containing this clause will only be applied to JSON topics.

26294: Topic type conversions

Topic views have a new 'type' clause which allows you to specify a topic type for the generated reference topic which may be different from the source topic type. If the selected source topic (or current transformed value) can be converted to the specified type, a reference topic of the specified type will be created; otherwise, there will be no reference topic. For a full specification of the supported conversions see the API documentation or the user manual.

Topics

22432: Session trees

A new 'session trees' feature is included in this release.

A session tree is a customized version of the topic tree presented to a session, adjusted by applying declarative branch mapping rules maintained by the server. Each session can be presented with a unique session tree based upon its session properties.

Session trees replace the previous 'routing topics' functionality but providing a much more powerful and flexible rules-based approach.

See the user manual and client API documentation for a full description of session trees.

Improvements in 6.7.0

C Client

24042: Close session using session ID

The C client can now close other sessions via the method diffusion_client_close_with_session which receives a sessionID in its parameters.

Client

22602: API support for creating and maintaining metric collectors

In previous releases, metric collectors could only be created and maintained using the Diffusion management console or JMX. There is now a new Metrics feature in the Client APIs that allows for creation and maintenance of metric collectors.

JavaScript Client

24858: Map non-string values to strings, for values of Options.properties

When setting session properties in diffusion.connect(), non-string values can be passed as property values.

Licensing

24000: Nightly license check expiry warning

Nightly license check now also warns about expiry.

Python Client

24430: Topic removal returns the number of topics removed

Topics.remove_topic now returns a TopicRemovalResult. The TopicRemovalResult.removed_count property returns the number of non-reference topics that were removed as a result of the operation.

26146: Support indefinite length CBOR encoding

This change implements indefinite encoding of CBOR, the preferred variant of the binary protocol used to communicate with the Diffusion server, for the Python module via an additional dependency called 'diffusion-cbor'.

To install from PyPi, proceed as before.

To install from the download package, do the following:

1. Unzip the Python client.
2. Unzip the Diffusion CBOR Python package corresponding to your platform (macOS, Windows or Linux) and run:
`pip install {path to python client wheel} --find-links {path to unpacked CBOR package)`

Other platforms are not yet supported.

27245: Local installation method for Python Client updated

Documentation now reflects the new local installation method.

Replication

27038: Avoid delta calculations when shipping large numbers of time series topic events to replica servers

In previous releases, if the cluster topology changed, replication of time series topics that have many events could cause the server to stall (with reports of busy multiplexer threads) due to the CPU cost of delta calculations. From this release, delta streams are no longer used when replicating large time series histories.

Deprecations in 6.7.0

.NET Client

27219: Callbacks deprecated

All methods that use callbacks have been deprecated at this release. Task based variants should be used in preference.

Java Client

26739: Callbacks are deprecated

All methods that use callbacks have been deprecated in this release. Equivalents that use CompletableFutures should be used instead.

26846: ClientControl.sessionIdFromString deprecated in favor of new Diffusion.sessionIdFromString


27204: ClientControl.setQueueEventHandler(QueueEventHandler) deprecated

The method ClientControl.setQueueEventHandler(QueueEventHandler) has been deprecated in favor of the newer setQueueEventHandler(QueueEventStream) method.

27205: ClientControl.setSessionPropertiesListener(SessionPropertiesListener) deprecated

The ClientControl.setSessionPropertiesListener(SessionPropertiesListener) method has been deprecated in favor of the newer setSessionPropertiesListener(SessionPropertiesStream) method.

27206: TopicControl.addMissingTopicHandler(String, MissingTopicHandler) deprecated

The TopicControl.addMissingTopicHandler(String, MissingTopicHandler) method has been deprecated in favor of the newer addMissingTopicHandler(String, MissingTopicNitificationStream) method.

27207: TopicControl.newSpecification deprecated

The TopicControl.newSpecification method has been deprecated in favor of the newer Diffusion.newTopicSpecification method.

Topics

26333: Routing topics are deprecated

Routing topics are deprecated at this release and will be removed in a future release. The more powerful SessionTrees feature introduced in this release should be used in their place.

Removals in 6.7.0

Client

23287: MessagingControl feature interface removed

The deprecated MessagingControl feature has been removed. All its functionality has been moved to the Messaging feature.

26331: Some unused TopicAddFailReason values have been removed

Unused TopicAddFailReason codes deprecated in Diffusion 6.2 or earlier have been removed from the client APIs.

Java Client

23283: com.pushtechnology.client.callbacks.TopicTreeHandler removed

The unused interface com.pushtechnology.client.callbacks.TopicTreeHandler has been removed.

Fixes in 6.7.0

.NET Client

27139: Credentials type is incorrect when authenticating

The .NET client was giving the wrong credentials type for custom credentials when authenticating. This has now been resolved.

Apple Client

26573: Apple client cannot connect securely to Diffusion Server via proxy

When the Apple client connected via secure WebSocket to a Diffusion server via a proxy, the connection would be downgraded to a non-secure WebSocket. This has now been fixed.

Client

27105: Applying JSON patch to a time series topic fails with server exception

The 'apply JSON patch' feature only works for JSON topics and should fail with an IncompatibleTopicException for any other type. However, if it was attempted for a time series topic with a JSON event type, a server side exception occurred and the correct exception was not reported to the client.

This has now been resolved. An attempt to apply a patch to a time series topic now results in the same exception being reported to the client as for other non-JSON topic types.

Console

26385: JSON data in time series history table still displays buffers

The Diffusion management console could previously display incorrect data in previews of past JSON time series topic values, when a 64 bit integer was present in the topic data. This has been fixed.

26450: Connection count on gateway tab can be misleading

The Diffusion management console could display confusing information about cluster servers when operating under a license with no connection limit. The console will now display "Unlimited" instead of "0" for the connection limit in this circumstance.

26607: Topic metric layout problem: wide tables overflow without providing usable scrollbar

The Diffusion management console tables for session and topic metrics could overflow the page horizontally on smaller displays. The tables are now horizontally scrollable within the page.

26652: Session properties do not appear

In Diffusion Management console releases prior to 6.6, the session browser view presented selected session properties when a particular session was selected. Due to a technical oversight, this feature was missing in 6.6. The feature has been restored, and all session properties for a particular session are now visible by selecting the session in the session browser view.

26653: Help text for $Principal and $Roles session properties is reversed

In the Diffusion management console, popover help text for the $Principal and $Roles session properties was shown for the wrong property. This has been corrected.

26983: Console presents inconsistent time zones in time series inspector

In the Diffusion management console, the time series inspector accepted time ranges in the local timezone, but displayed the timestamps of results in the UTC timezone. Event timestamps are now displayed in the local timezone, along with their offset from UTC.

Demos

22654: Racing Cars demo hangs when started without Diffusion server

Racing car demo now displays meaningful error when unable to connect to a Diffusion server.

Federation

27063: Remote server reconnection has memory leak

Repeated reconnections of remote servers could lead to a memory leak. This has now been resolved.

Java & Android Client

26318: Header parsing during proxy authentication is not case insensitive

An issue where headers such as Keep-Alive and Connection were not checked ignoring case during proxy authentication has been fixed.

JavaScript Client

26226: JSON deltas fail with indefinite length arrays and objects

JSON deltas failed when the JSON was encoded using CBOR with indefinite length arrays and objects. Now JSON deltas work with definite and indefinite length encoded CBOR.

26439: Deregistration callbacks return Promises but this is not reflected in the TypeScript API or documentation

Deregistration callbacks in MissingTopicHandler and SessionPropertiesListener return a Promise that resolves when the listener has been deregistered. This is now reflected in the documentation and in the TypeScript API.

26441: DataType.writeValue can accept null but this is not reflected in the TypeScript API

writeValue for primitive and JSON datatypes can accept null as a value. This is now reflected in the TypeScript API

26662: int64 docs reference overflowing number

The documentation of int64 referenced a number that would cause an overflow. It has now been replaced with a number that fits into a signed 64-bit integer.

26799: JSON.jsonDiff(original) does not take arbitrary objects as documented

Previously, JSON.jsonDiff(original) did not take arbitrary plain objects as documented. This has been fixed.

26860: Subscribing and appending to time series topics fails with shared sessions

There was a bug preventing plain data value streams to subscribe to time series topics when using shared sessions. This has been resolved.

26981: Error when trying to fetch time series topics using AnyDataType

FetchRequest threw an error when trying to obtain values from time series topics using AnyDataType. Now it will return the time series event on the FetchResult.

27136: Topic properties documentation issues

The documentation of topic properties in TopicSpecification contained incorrect entries and has been fixed.

MQTT

26694: Improve error handling of MQTT payloads that can't be mapped to Diffusion topic values

In previous releases, if an MQTT session attempted to publish a payload that could not be converted to an appropriate topic value, the server would log a stack trace and terminate the session. For example, publishing a string that could not be converted to an integer to an INT64 topic, or publishing a non-JSON string to a JSON topic.

From this release, the server handles these scenarios more gracefully, logging a warning message, and rejecting the publication with the MQTT error code 0x83 (implementation-specific error).

Python Client

26980: mypy objects to missing mock type hints in Python 3.7/3.8 on some platforms

This change addresses how type checking in the build process handles missing type stubs. It now pulls in any missing type stubs automatically where available, and ignores the remainder. There were no changes in shipping code in the latest version.

Replication

26370: PUSH-000753 not logged when quorum met

Due to a bug introduced in Diffusion 6.6, the server did not log a message when quorum was met. This been rectified in this release.

26407: Cluster routing could fail to find a handler hosted by a peer

When a server is part of a cluster, if it has no local handler for various types of event such as messages, missing topic notifications, and authentication requests, but a peer server does, the event will be forwarded to the peer for processing.

Due to a bug in previous releases, some requests would not be forwarded even though a peer with a suitable handler existed. The bug has been fixed in this release.

Security

26950: TOPIC_OWNER topic property not respected when a session changes its principal

When a session re-authenticates using the "change principal" API, the server re-evaluates its subscriptions. In previous releases, the re-evaluation failed to take account of the TOPIC_OWNER topic property. This bug has been fixed in this release. Now a session's subscriptions can change if it re-authenticates using a different principal, so that it gains or loses ownership of a topic.

Server

27096: Thread deadlock between DefaultPicoContainer and RemoteServerLinkManagerImpl

When using remote topic views, a thread deadlock could occur. This would manifest in a thread dump as a deadlock between DefaultPicoContainer and RemoteServerLinkManagerImpl. This has now been resolved.

System Monitoring/Statistics

26581: JMX service binding to all interfaces, despite configuration

Diffusion 6.5.0 added support for JMX over TLS, and made this the default configuration.

Due to a bug, when configured for TLS, the JMX service always bound to all local network interfaces regardless of whether a specific interface was specified by the host parameter in Management.xml. The bug has been fixed in this release, and the host parameter is now respected.

Topic Views

26584: Range query fails on time series reference topic created from a remote topic view

A query against a time series reference topic created from a remote topic view would fail, indicating that the topic was not a time series topic. This problem has now been resolved.

27134: NullPointerException evaluating a Time Series JSON topic with a view that contains an insert clause

A topic view could incorrectly select a time series JSON topic and result in a NullPointerException at the server if the view contained both a 'scalar' or 'as value' clause and a clause that applies only to JSON topics (such as insert or expand).
This has now been resolved.

Topics

26973: Time series range query returns incorrect values

In some situations, a time series topic range query could return incorrect values. This could be seen in the console or in range queries issued by clients. This has been resolved in this release.

Known Issues

Console

27844: Console shows fractional users connected

The concurrent clients and client connections graphs on the console can display fractional users.

Topic Views

27594: Reference topics retained by 'preserve topics' are not persisted across server instances or cluster

A new 'preserve topics' clause was introduced to topic views in release 6.6. This clause means that reference topics created by a view (that have a path dependent upon the source topic value) are retained until the source topic is removed or the topic view is removed. Though this is true in the context of a single server instance, it is not the case if the server is restarted as all such topics created during the previous server instance will be lost. It is also not the case if a new server enters a cluster as the new server will only have reference topics generated from the point in time where it joined the cluster and will not reflect reference topics previously created within other cluster peers.

This issue occurs because reference topics are not persisted, either to file or across the cluster.

This issue will be resolved in a future release.