Diffusion Cloud 6.10 Release Notes

6.10.2 (14 February 2024)

Fixes in 6.10.2

.NET Client

DIF-1177: Client ignores maximum message size set by user

The Dotnet client ignored the maximum message size set by the user. This has been fixed.

DIF-1196: GetSessionProperties does not throw NoSuchSessionException

Calling GetSessionProperties for a session that has been closed did not throw NoSuchSessionException. This has been fixed.

DIF-238: Authentication Control example throws exception before completing

If SetAuthenticationHandlerAsync is called with await then creating a session throws SessionEstablishmentTransientException before the authenticator is called. This has been fixed.

DIF-63: SessionClosedException thrown when not in closed state

If timeout occurs during reconnection the client can hang. This has been fixed.
A connection attempt does not timeout if there is no response from the server. This has been fixed.
Connection and reconnection failure will transition the session state to CLOSED_FAILED.

Apple Client

DIF-779: Missing Session State Listener

The Session State Listener was missing from PTDiffusionSessionConfiguration. This issue has been fixed.

DIF-797: Invalid URL string when opening a session in the Apple SDK raises uncaught NSInvalidArgumentException

When an invalid URL string is passed when opening a session a NSInvalidArgumentException is raised.
In Objective-C we can trap it, but in Swift it’s not possible resulting in an uncaught exception that crashes the application. This has been fixed.

DIF-847: Reconnection Timeout closes successfully recovered connection

When a successful reconnection occurred, the reconnection timeout was not properly cleared and resulted in a disconnection. This issue has been resolved.

Console

DIF-178: Confusing displays for licences without connection limits

Licences without hard connection limits could previously display an incorrect message on the network overview tab of the console.

DIF-180: Topic views errors do not show the correct column position

The console topic view editor provided incorrect column information when presenting syntax error feedback for topic views.

DIF-792: Metrics table can fail to pad collector metrics correctly

When a metrics table is displaying metrics from multiple collectors with different numbers of dimensions, rows for those collectors with fewer dimensions are incorrectly shifted left by one or more columns.

Logging

DIF-834: Existing PUSH message numbers changed at 6.10

Some of the PUSH message numbers, as documented in the user manual changed their meaning between 6.9 and 6.10. For example, PUSH-000882 had a different meaning at 6.10 than at 6.9. All numbers from PUSH-000880 were incorrectly assigned.
This has now been resolved.

Python Client

DIF-1162: ListEncoder.read uses Byte for length instead of Int32

This fix resolves an issue with list serialisation in the Python Client, due to the representation of length being given by a byte rather than a 32 bit integer, preventing lists larger than 255 items being serialised or deserialised correctly. The features formerly affected include: Topic/Session Metric Collector lists, Conjunction Constraints lists, Branch Mapping lists, Range Query event lists, Error Report lists and Session Tree branch lists at the time of discovering this bug.

DIF-30: No support for Python 3.10-12

The Python client now supports Python versions 3.8-3.12 on MacOS (arm64 and x86_64), Windows and any x86_64 Linux distribution supported by the Manylinux project (https://github.com/pypa/manylinux).

Server

DIF-1023: Corruption of client queue index can cause unnecessary CPU use

Due to a bug in previous releases, long lived sessions that subscribed to many different topics could perform unnecessary processing. The bug has been fixed in this release.

DIF-849: The UNSUBSCRIBE conflation policy does not handle branch mapping subscriptions correctly

Due to a bug in previous releases, the UNSUBSCRIBE conflation policy did not correctly handle branch mapping subscriptions. This has been fixed in this release.

DIF-917: Server can fail to accept updates at a high rate

Due to a bug in previous releases, a server could fail to accept updates at a high rate (more than tens of thousands of updates a second). The bug has been fixed in this release.

Session Trees

DIF-608: Range queries fail on timeseries topics exposed through session trees

An attempt to perform a range query on a topic exposed by session trees would fail with "Range query failed because topic 'xxx' does not exist.". This has now been resolved so that range queries can be performed on session tree time series topics.

6.10.1 (12 October 2023)

Fixes in 6.10.1

.NET Client

DIF-36: Unable to use 'and' method with two value constraints

When using the update constraints API there was an issue where it was not possible to use 'and' composition with two value comparison constraints. For example, it was not possible to say 'value GT 5' and ' value LT 10'. This has now been resolved. The relaxation of validation does mean that it is now technically possible to define a constraint that could never evaluate to true - for example 'value EQ 5' and 'value NE 5' so it is up to the user to ensure constraints are possible.

Apple Client

DIF-36: Unable to use 'and' method with two value constraints

When using the update constraints API there was an issue where it was not possible to use 'and' composition with two value comparison constraints. For example, it was not possible to say 'value GT 5' and ' value LT 10'. This has now been resolved. The relaxation of validation does mean that it is now technically possible to define a constraint that could never evaluate to true - for example 'value EQ 5' and 'value NE 5' so it is up to the user to ensure constraints are possible.

C Client

DIF-36: Unable to use 'and' method with two value constraints

When using the update constraints API there was an issue where it was not possible to use 'and' composition with two value comparison constraints. For example, it was not possible to say 'value GT 5' and ' value LT 10'. This has now been resolved. The relaxation of validation does mean that it is now technically possible to define a constraint that could never evaluate to true - for example 'value EQ 5' and 'value NE 5' so it is up to the user to ensure constraints are possible.

Java Client

DIF-36: Unable to use 'and' method with two value constraints

When using the update constraints API there was an issue where it was not possible to use 'and' composition with two value comparison constraints. For example, it was not possible to say 'value GT 5' and ' value LT 10'. This has now been resolved. The relaxation of validation does mean that it is now technically possible to define a constraint that could never evaluate to true - for example 'value EQ 5' and 'value NE 5' so it is up to the user to ensure constraints are possible.

JavaScript Client

DIF-20: Javascript API has no mechanism for setting trusted certificates

Additional options have been added to diffusion.connect, allowing modification of the underlying TLS configuration. The new options are passed directly to tls.createSecureContext()

DIF-36: Unable to use 'and' method with two value constraints

When using the update constraints API there was an issue where it was not possible to use 'and' composition with two value comparison constraints. For example, it was not possible to say 'value GT 5' and ' value LT 10'. This has now been resolved. The relaxation of validation does mean that it is now technically possible to define a constraint that could never evaluate to true - for example 'value EQ 5' and 'value NE 5' so it is up to the user to ensure constraints are possible.

DIF-479: Shared session has initialising state immediately after connecting

Shared sessions would return the connected session before updating the state to connected. This could lead to a new session to appear to still be in the `initialising` state. The order of internal state updates has been corrected. A new session will now reliably be in `connected` state.

Replication

DIF-149: Error while handling a multiplexer event - IllegalStateException: Has already connected

In previous release, when new servers joined a cluster, a server could log a stack trace with the error message "IllegalStateException: Has already connected". This is harmless, but overly concerning. From this release, the error message will no longer be logged.

DIF-404: Thread gets blocked when another cluster member is killed

Fixed a rare issue with non-blocking replication that could cause a server to block new topic updates indefinitely.

Server

DIF-513: Update CAS failure after killing cluster member

When using a cluster with a quorum a Diffusion instance could reject further updates if another instance was restarted. This issue has now been resolved.

DIF-556: Multiplexers can encounter errors when topics are updated during partition recovery

In a rare situation during cluster reconfiguration an update to a replicated topic could fail to be sent to clients. This has been fixed.

6.10.0 (30 August 2023)

Improvements in 6.10.0

.NET Client

FB-29884: Remote Server support for reverse connections

The remote server feature now supports the creation of remote servers for reverse connections (introduced in release 6.9), that is connections initiated from a primary server rather than from a secondary server.

FB-30210: SessionIdFromString method

The .NET client can now create a session ID from a string with the SessionIdFromString method.

Apple Client

FB-29881: Remote Server support for reverse connections

The remote server feature now supports the creation of remote servers for reverse connections (introduced in release 6.9), that is connections initiated from a primary server rather than from a secondary server.

C Client

FB-29766: Removal of unused message types

Several unused and deprecated message types have been removed from the C Client

FB-29882: Remote Server support for reverse connections

The remote server feature now supports the creation of remote servers for reverse connections (introduced in release 6.9), that is connections initiated from a primary server rather than from a secondary server.

Client

FB-28527: Ability to retrieve metrics using the API

At this release, there is a new API for retrieving metrics. This is enabled in the Metrics feature. This is available in the Java, .NET, and JavaScript SDKs.

FB-28682: Update Stream Recovery

The update stream builder now allows a RecoverableUpdateStream to be created that can be used to recover automatically from transient clustering exceptions.

FB-28999: TopicAddFailReason INCOMPATIBLE_PARENT and IncompatibleParentTopicException Removed

The deprecated TopicAddFailReason.INCOMPATIBLE_PARENT and associated IncompatibleParentTopicException have been removed at this release from the Java SDK. Equivalents have also been removed from all other SDKs.

FB-29000: TopicPermission and associated methods removed

TopicPermission was deprecated at release 6.5 in favor of PathPermission. It has been removed at this release along with all associated methods.

FB-30017: Conditional update constraints

At this release the UpdateConstraint API has been extended to allow for conditional constraints which can check the value of a topic or JSON field using operators like GT (greater than), LT (less than) etc. This functionality is available in all client SDKs other than Python (for a future release).

FB-30328: OR operator for Update Constraints

It is now possible to form compound update constraints using an 'or' operator as well as 'and'. This functionality is available in all SDKs except Python at this release.

Console

FB-30139: Improvements to permissions handling in the console

The Diffusion management console now behaves more predictably and provides more user feedback when attempting to browse a topic tree with restricted permissions at the root.

Gateway

FB-28152: REST API to Manage Gateway Framework

In this release, we have introduced the Gateway Control REST API, a powerful feature that allows for the management and monitoring of Gateway applications through REST API calls. Prior to this release, Gateway applications could only be managed through the Diffusion management console. With the new REST API, all operations previously available exclusively in the console can now be performed using REST calls. For more information, please refer to the updated manual.

Java & Android Client

FB-30025: Java Client and Diffusion server now use HTTP/1.0 standard headers

The Java client and the Diffusion server now follow HTTP/1.0 standards, including an ASCII space following the colon in HTTP headers. Other Diffusion clients already follow this practice and are unchanged.

Python Client

FB-26147: Topic update using constraints

The Python SDK is now capable of updating topics using update constraints

FB-29743: Make serialiser resolution order-invariant, test without caches


Replication

FB-30726: Hazelcast upgraded to 5.2.4

Due to security vulnerabilities in earlier versions, Hazelcast has been upgraded to release 5.2.4.

System Monitoring/Statistics

FB-28456: Metrics Improvements for 6.10


Topic Views

FB-29798: Floating point support in topic views

Topic views now support the use of floating point numbers in all clauses, particularly in process calculations, etc. This includes the ability to specify the scale (number of places after the decimal point) and the rounding policy to use for calculation results.

For full details see the Topic Views page in the user manual.

Topics

FB-29802: Routing Topics Removed

ROUTING topics, which were deprecated in release 6.7, have been removed at this release. There is no longer support for routing topics in the server. Users who were using routing topics should now use the more powerful 'Session trees' feature.

Fixes in 6.10.0

.NET Client

FB-29978: NoReconnection does not correctly disable reconnection

Addresses an issue where setting NoReconnection on a .NET session did not disable reconnection. This has now been resolved.

FB-30284: .NET client incorrectly accepts empty string as valid JSON

The dotnet client incorrectly accepted empty strings as valid JSON. This has now been fixed.

FB-30740: Unable to connect successfully through a load balancer

Addresses an issue where the .NET client was unable to connect through a GCP (Google Cloud Platform) load balancer.

Apple Client

FB-30396: Frequent crash in Diffusion iOS client library

We have addressed a bug that crashed the Diffusion Apple Client in PTDiffusionRequestStreamRegistry.
This issue is now resolved.

C Client

FB-29801: Session properties listener seg faults when an update event is received

Addresses an issue where a C client application could crash (segfault) if a session close event was received in a session properties listener.

Federation

FB-29642: Remote server connections can block indefinitely

Networking issues between clustered Diffusion instances could cause the server to stall.

This has been fixed in this release.

Java & Android Client

FB-30219: Some API calls return ExecutionException rather than CompletionException

All API calls that return a CompletableFuture should fail exceptionally with a CompletionException and not ExecutionException. However, at the previous release, there were some cases where this was not true including:-

1) When UpdateStream methods were returning InvalidUpdateStreamException.

2) Calls to Metrics.removeSessionMetricCollector, Metrics.removeTopicMetricCollector, and Metrics.putTopicMetricCollector.

JavaScript Client

FB-29448: Transport level error does not notify session establishment promise

In Safari 16.0, there is an experimental feature called "NSURLSession WebSocket". With this enabled, any JS API session failure will terminate the transport level websocket connection, resulting in errors. However, such errors were not being propagated back to the client's session establishment promise, leading to session establishment never completing. This problem may also exist in other browsers.
The problem has been resolved in this release.

FB-29729: MutableRecordModel.add not working as expected

MutableRecordModel.add was not working as expected and records were not added. This has been fixed.

FB-30327: Incorrect close reason type on SessionPropertiesListener.onSessionClose

The TypeScript definition of SessionPropertiesListener.onSessionClose() did not specify the correct type for the close reason. A new ClientCloseReason type has been introduced to the signature of the callback.

Logging

FB-30400: Compaction corruption messages logged at ERROR level

Log messages that detect a persistence file corruption were previously being logged at ERROR level, even though the problems are automatically corrected. These messages are now logged at WARN level and have PUSH code messages.

Persistence

FB-30715: Topic persistence operations could fail silently

Fixed an issue that could cause commit of topic operations to fail silently.

Replication

FB-29959: Cluster repartitioning during a request to add a topic can cause a the request to hang

If cluster repartitioning occurred while a client was adding a topic then instead of the client being notified with an exception the operation would never complete. This has been fixed.

FB-30611: If cluster topology changes during topic removal, topics can be left in a state where they can never be removed

If the cluster topology changes, that is, a server joins or leaves the cluster, in-flight operations can fail with a transient CLUSTER_REPARTITION error. Due to a bug introduced in Diffusion 6.5, topic removal operations that fail if the cluster topology changes can leave the topics in an invalid state such that they can never be removed.

The bug has been fixed in this release.

FB-30697: Failure to start connector because readiness conditions not triggered

Fixed an issue that could result in Diffusion instances joining a cluster failing to reach readiness.

FB-30701: Problems when restarting a cluster member with an out of date persistence file

Fixed an issue that could cause unexpected restarts when an instance with an out-of-date persistence file joined a cluster.

Security

FB-29728: Users with no permissions can view the server logs in the console

Users without adequate permissions were able to view server logs in the Diffusion Management Console. This has now been resolved.

FB-30250: A path permission assignment for a role would sometimes not override permissions inherited from a parent path

Due to a bug in previous releases, if a security role had path permission assignments for paths X and Y, where X is a parent path of Y, the path permission assignments for Y were sometimes not applied correctly.

The bug has been fixed in this release.

Server

FB-29670: Multiplexer stuck processing selection of a missing topic notification handler

Due to a concurrency bug in previous releases, the server could stall while trying to select a control session to handle a request.

The bug has been fixed in this release.

FB-30364: StringIndexOutOfBoundsException in PathSelector.confirmSelects

In certain circumstances, the Diffusion server could report a StringIndexOutOfBoundsException in PathSelector.confirmSelects. This has now been resolved.

Topic Views

FB-29995: Topic view INSERT can incorrectly insert the value of a topic with DONT_RETAIN_VALUE=true

Due to a bug in previous releases, a topic view INSERT transformation could insert a stale value of a stateless topic (that is, a topic with its DONT_RETAIN_VALUE property set to true). This would happen if the topic view was added after the stateless topic, and there were other topic views with topic selectors that referenced the stateless topic.

The bug has been fixed in this release. An INSERT transformation will never insert the value of a stateless topic.

FB-29999: Reference topics can incorrectly use the stale value of a topic with DONT_RETAIN_VALUE=true

Due to a bug in previous releases, topic view could create reference topics using a stale value of a stateless topic (that is, a topic with its DONT_RETAIN_VALUE property set to true). This would happen if the topic view was added after the stateless topic, and there were other topic views with topic selectors that referenced the stateless topic.

The bug has been fixed in this release.

FB-30686: Mapping single value topic to time series topic does not respect specified retained range

Using topic views to map a single value topic to a time series topic would always assume the default retained range (10) regardless of what was specified in the with properties clause. This was because the retained range for time series reference topics is always limited to the source retained range so that cross cluster inconsistencies do not occur. This has now been changed so that when mapping a single value topic to a time series topic any specified retained range is respected rather than assuming the default.
However, there is a separate known issue relating to this type mapping in that using a single value topic mapping to a time series target can lead to inconsistent views of the target topic events across a cluster and therefore such a mapping is not suitable for use in a clustered environment. Even in a single server environment it is worth noting that the retained event range of the target time series topic will not be restored in server restart.

Topics

FB-29971: Subscriptions fail to be re-evaluated if topic selections match topics in the source branch of a removed branch mapping

Due to a bug in previous releases, if a subscription to a topic was redirected by a branch mapping, and the branch mapping was removed, the session may not be resubscribed to a topic at the original topic path. The bug has been fixed in this release.

Known Issues

Topic Views

DIF-166: Reference topics retained by 'preserve topics' are not persisted across server instances or cluster

A new 'preserve topics' clause was introduced to topic views in release 6.6. This clause means that reference topics created by a view (that have a path dependent upon the source topic value) are retained until the source topic is removed or the topic view is removed. Though this is true in the context of a single server instance, it is not the case if the server is restarted as all such topics created during the previous server instance will be lost. It is also not the case if a new server enters a cluster as the new server will only have reference topics generated from the point in time where it joined the cluster and will not reflect reference topics previously created within other cluster peers.
This issue occurs because reference topics are not persisted, either to file or across the cluster.

DIF-167: Restrictions on mapping single value topics to time series reference topics

The ability to specify a target type in a topic view was introduced in release 6.7.0.
When using this feature to map a single value topic to a time series topic there are the following restrictions.
1) The retained events in the target topic are not replicated across the cluster (as reference topics are not replicated). This means that a new server joining the cluster will not have the same number of retained events as other cluster members. For this reason mapping single value to time series topics should not be used in a clustered environment.
2) Retained events in the target topic are not persisted therefore when a server is restarted the target time series will initially start with a single event and will only grow as the source topic is updated.