Diffusion 6.6 Release Notes

6.6.1 (5 May 2021)

Fixes in 6.6.1

Apple Client

26572: Apple client cannot connect securely to Diffusion server via proxy

When the Apple client connected via secure WebSocket to a Diffusion server via a proxy, the connection would be downgraded to a normal WebSocket. This has now been fixed.

Java & Android Client

26359: Header parsing during proxy authentication is not case insensitive

An issue where headers such as Keep-Alive and Connection were not checked ignoring case during proxy authentication has been fixed.

JavaScript Client

26304: JSON delta fails with indefinite length arrays and objects

JSON deltas failed when the JSON was encoded using CBOR with indefinite length arrays and objects. JSON deltas now work with definite and indefinite length encoded CBOR.

26549: Deregistration callbacks return Promises but this is not reflected in the TypeScript API or documentation

Deregistration callbacks in MissingTopicHandler and SessionPropertiesListener return a Promise that resolves when the listener has been deregistered. This is now reflected in the documentation and in the TypeScript API.

26552: DataType.writeValue can accept null but this is not reflected in the TypeScript API

writeValue for primitive and JSON datatypes can accept null as a value. This is now reflected in the TypeScript API.

Logging

26461: Some warnings were logged without PUSH codes

Due to a bug introduced in Diffusion 6.6 Preview 2, warnings about failures to update topics were logged without the associated PUSH code (PUSH-000464, PUSH-000843, or PUSH-000844). The bug has been fixed in this release.

Replication

26372: PUSH-000753 not logged when quorum met

Due to a bug introduced in Diffusion 6.6, the server did not log a message when quorum was met. This been rectified in this release.

26408: Cluster routing could fail to find a handler hosted by a peer

When a server belongs in a cluster, if it has no local handler for various types of event such as messages, missing topic notifications, and authentication requests, but a peer server does, the event will be forwarded to the peer for processing.

Due to a bug in previous releases, some requests would not be forwarded even though a peer with a suitable handler existed. The bug has been fixed in this release.

System Monitoring/Statistics

26585: JMX service binding to all interfaces, despite configuration

Diffusion 6.5.0 added support for JMX over TLS, and made this the default configuration.
Due to a bug, when configured for TLS, the JMX service always bound to all local network interfaces regardless of whether a specific interface was specified by the host parameter in Management.xml. The bug has been fixed in this release, and the host parameter is now respected.

Topic Views

26452: Reference topics not removed with source topic if chained view has source value directives

If a source value directive was used in a topic view that operated upon topics downstream from another topic view, the reference topics would not be removed when their source topic was removed. This problem has now been resolved.

26454: A throttled topic view can produce an invalid delta stream

A topic view with a throttle clause could produce a corrupt delta stream. The bug has been fixed in this release.

26471: Processing errors when chaining a topic view using "preserve topics" to other topic views

Diffusion 6.6.0 added a "preserve topics" topic view option which can be used in conjunction with the scalar and expand path mapping operators. Due to a bug in the implementation, processing errors could occur when chaining the output of such a topic view to other topic views. The bug has been fixed in this release.

26590: Range query fails on time series reference topic created from a remote topic view

A query upon a time series reference topic created from a remote topic view would fail indicating that the topic was not a time series topic. This problem has now been resolved.

6.6.0 (19 March 2021)

New features in 6.6.0

Adapters

24846: Visualisation of Adapters in the Management Console

It is now possible to visualize Diffusion adapters (currently Kafka and JMS) in the management console. All adapters connected to the server (or cluster) can be viewed and will show their current status. It is also possible to view a connected adapter's configuration. Once an adapter has connected, a record of it is persisted so that even when not connected it can be seen as a disconnected adapter in the console.
In future releases, this feature will be extended so that you will be able to configure and control adapters from the console.

Console

14551: Console cluster view

The Diffusion management console now provides an overview of the cluster which the server is part of, including status information on connected gateway clients such as the Kafka and JMS adapters. The new view can be found in the Network tab.

24524: Single sign on support for management console

The Diffusion management console can now be integrated with third party single sign-on systems. For more information, see "Configuring the Diffusion management console" in the Administrator's guide.

JMS Adapter

24814: New version of JMS adapter

A new version of the JMS adapter is included in this release.

The format of the adapter configuration file has changed from XML to JSON.

The adapter now supports Diffusion topic subscription, which enables updates to Diffusion topics to be sent to JMS destinations. The adapter can be visualized and monitored in the Diffusion management console. Request-response messaging functionality has been removed.

For more details on configuring and running the adapter, see the manual.

JavaScript Client

23227: Javascript Client - dynamic library loading

In addition to the regular JavaScript Client bundle, there is now a collection of bundles that package the client in a modular way. The core client bundle is reduced in size to reduce loading times. All additional client features can be loaded dynamically after a session has connected.

Kafka Adapter

26102: Kafka adapter

This release includes a new Kafka adapter for exchange of data between an Apache Kafka cluster and a Diffusion server. The adapter can be used to translate Diffusion topics to Kafka, or Kafka topics to Diffusion, or both at the same time.

Configuration options enable you to rename and restructure the translated data. Monitoring is available via Prometheus. The adapter can be visualized and monitored in the Diffusion management console as well.

See the manual for details.

MQTT

23625: MQTT

The Diffusion server now accepts connections from clients using the MQTT 5.0 protocol.

MQTT clients can publish and subscribe to Diffusion topics. Both the TCP and WebSocket transports can be used. (Currently Diffusion Cloud only supports the WebSocket transport.)

MQTT sessions are fully integrated with Diffusion security. Authorization rules apply to publication and subscription​. Connections can optionally be secured with TLS.

Python Client

24830: Python SDK Client (beta version)

A full native Python SDK Client has been added to connect to Diffusion.
Its current capabilities are:
- Session:
- Establish a session
- Send user ping
- Receive system ping
- Messaging:
- Register request handler
- Send request to path
- Send request to session
- Send request to filter
- Register filter message stream
- Topics:
- Topic subscriptions
- Topic creation
- Topic creation with initial topic value
- Topic removal
- Topic stream handlers

Topic Views

20476: Topic ownership enhancements to topic view security

At this release, there have been several enhancements to the way in which the topic OWNER property interacts with topic views and reference topics created by them:

Topic views now respect the OWNER property when evaluating source topics. This means that a source topic will now be selected for evaluation by a topic view if the principal of the topic view is the same as the owner of the source topic.

The OWNER property (if set) is now copied from source topics to all reference topics that are derived from them. This means that if ownership is granted to a topic, it is also granted to all reference topics derived from it.

Both the topic view principal and the OWNER property (if set) are now treated as owners of a reference topic. This means that full access is given to the principal of the view that created a reference topic, as well as any OWNER principal specified.

20771: New 'separator' option in topic view specifications

The topic view specification language has a new option clause called 'separator'. You can use this to specify a separator string to replace the path separator '/' in extracted scalar values that are used to form all or part of the generated reference topic path. If specified, it will apply to all 'expand' or 'scalar' directives or 'insert' options used within the view.

24034: Topic views: preserve topics clause

A new clause called 'preserve topics' has been added to the topic view specification Domain Specific Language.

Normally when a topic view is evaluated against a source topic, when it is updated, if there are directives within the specification that derive the path of the reference topics from the source value (such as scalar and expand), any reference topics created previously that no longer have mappings from the new value are removed.

If the 'preserve topics' clause is used, all reference topics created by a view are preserved until the source topic itself is removed (or they are removed by higher precedence views or non-reference topics).

24137: Topic view inserts

The new topic view inserts feature allows values from topics other than the source topics being evaluated for a topic view to be inserted into the values of reference topics. This is enabled by the new 'insert' clause in the topic views Domain Specific Language. See the documentation for full details.

Improvements in 6.6.0

.NET Client

16267: Async methods added to IClientControl

A method for closing a client session which returns a Task is now available. This accepts a sessionId. If the identified session was closed, the operation completes successfully, and the Task result is null.

A method for control client queue conflation which returns a Task is now available. This accepts a sessionId and a boolean value to enable or disable conflation for the session. If the conflation policy was updated for the identified session, the operation will complete successfully, and the Task result will be null.

22422: Task version of getSessionProperties

A method for accessing session properties which returns a Task is now available. This accepts a sessionId and collection of required properties and returns a Task containing a dictionary of the requested properties for the client identified by the sessionId.

23858: Value streams now propagate events from subscribed time series topics with a compatible event type

Previously, events for time series topics were only received by time series event streams added with AddTimeSeriesStream. This has been changed so that all streams will emit the topic value without the related time series metadata, if they are compatible with the event type.

24428: Removing topics returns the number of topics removed

ITopicControl.RemoveTopicsAsync now returns a Task bearing an ITopicRemovalResult. Using its RemovedCount property returns the number of non-reference topics that were removed as a result of the operation.

Android Client

22693: Android jar no longer includes SLF4J API

The Diffusion Android client now no longer provides SLF4J API internally as part of the client. Instead, the client now requires an application dependency on SLF4J API. Specify this in the Gradle file you're using to build your application.

Apple Client

22365: ClientControl can now close other sessions and change client queue conflation of other sessions

Client control feature is now capable of closing session via sessionID or session properties filter, enabling or disabling the client queue conflation via sessionID or session properties filter and set queue event listeners.

22878: Filtered request callback is no longer a stream

PTDiffusionSessionResponseStreamDelegate methods diffusionStream:didFailWithError: and diffusionDidCloseStream: are now deprecated and will be removed in a future release.

23859: Value streams now propagate events from subscribed time series topics with a compatible event type

Previously, events for time series topics were only received by time series event streams added with addTimeSeriesStream.
This has been changed so that all streams will emit the topic value without the related time series metadata, if they are compatible with the event type.

24426: Removing topics returns the number of topics removed

PTDiffusionTopicControlFeature.removeDiscreteWithTopicSelectorExpression has been deprecated in favour of removeTopicsWithTopicSelectorExpression which returns a PTDiffusionTopicRemovalResult in the completion handler, containing the number of topics removed with the command.

C Client

23860: Value streams now propagate events from subscribed time series topics with a compatible event type

Previously, events for time series topics were only received by time series event streams added with add_time_series_stream.
This has been changed so that all streams will emit the topic value without the related time series metadata, if they are compatible with the event type.

24257: APR library update

Following the GLIBC update to v2.32, APR and APR-Utils libraries have been updated to 1.70 and 1.6.1 respectively.

24427: Removing topics to return the number of topics removed

TOPIC_REMOVAL_PARAMS_T.on_removed now returns a DIFFUSION_TOPIC_REMOVAL_RESULT_T instead of a SVC_TOPIC_REMOVAL_RESPONSE_T.

You can now call diffusion_topic_removal_result_removed_count on the response to return the total number of topics removed by topic_removal

Client

15926: Session properties filter: IN operator

An 'in' operator has been added to session filters to allow for filtering based on an item being present in a list.

20951: MissingTopicNotification processing has been simplified

Missing topic notification handling has been simplified.

Previously, subscription requests would not be processed until the server received a response from the missing topic handler. From this release, subscription requests are processed immediately by the server, without waiting for the handler. The proceed() and cancel() methods of MissingTopicNotification no longer have any effect and are deprecated.

A missing topic handler can no longer cancel a subscription. A supported alternative is to use the SubscriptionControl feature to issue a compensating unsubscribe operation.

21340: Update streams now respect topic conflation settings and can be used to update topics with DONT_RETAIN_VALUE=true

Update streams now support updating topics with the DONT_RETAIN_VALUE property set to true. Update streams now update these topics by sending whole values to the server instead of deltas.

When creating a new stream, conflation of pending values is now applied based on the conflation property of the topic.

23010: TopicUpdate feature can now be used to update time series topics

Topic update methods can now be used to update time series topics. This allows for creating a topic with an initial value, update constraints and delta streams.

23111: Fetch can retrieve the latest value from a time series topic

Fetch methods now return the latest value from a time series topic if the data type passed as a parameter in the fetch method matches or is compatible with the time series topic event's data type.

24493: New 'has' operator for session property queries

There is a new 'has' operator in the session properties query language which allows session filters to check if a property is present for a session.

Console

19345: Time series topic query and visualization

The Diffusion management console now provides an interface to fetch the retained previous values of time series topics. In addition, numeric time series topics are displayed graphically.

23469: Additional sorting and filtering controls for large configuration tables

Several tables in the Diffusion management console which can display many items, for example topic views, now have improved sorting and filtering controls to make locating individual items easier.

23854: Session view provides access to all session properties

The session browser of Diffusion management console now shows all session properties including all fixed properties. Expand a session in the table to see this information.

24766: Cluster-aware session and topics metrics

The Diffusion management console now allows users to view the session and topics metrics from each member of a Diffusion cluster, without having to switch consoles. An aggregated view of the whole cluster's metrics is also provided.

Environment

24663: Diffusion start scripts now allow setting of environment variables in a parent wrapper script

Diffusion start scripts now allow for the setting of environment variables in a parent wrapper script.

Java & Android Client

19053: Removing topics returns the number of topics removed

TopicControl#removeTopics now returns a CompletableFuture bearing a TopicRemovalResult. Calling getRemoveCount returns the number of non-reference topics that were removed as a result of the operation.

22818: New SessionEstablishmentTransientException from SessionFactory.open

A new exception called SessionEstablishmentTransientException has been introduced which can be returned from SessionFactory.open. This exception indicates a transient failure and the client application can reasonably retry the connection. The existing ServerInitializingException is now a subclass of the new exception.

23832: Value streams now propagate events from subscribed time series topics with a compatible event type

Previously events for time series topics were only received by time series event streams added with addTimeSeriesStream. This has been changed so that all streams will emit the topic value without the related time series metadata, if they are compatible with the event type.

Java Client

24481: Datatypes#forClass now returns the binary datatype when called on the Bytes class

In previous releases, Diffusion.datatypes().forClass(Bytes.class) threw IllegalArgumentException.class. From this release, Diffusion.datatypes().forClass(Bytes.class) returns the binary datatype. As a consequence of this change, Bytes instances can now be passed through the messaging API.

JavaScript Client

23857: Value streams now propagate events from subscribed time series topics with a compatible event type

Previously events for time series topics were only received by time series event streams added with addTimeSeriesStream. This has been changed so that all streams will emit the topic value without the related time series metadata, if they are compatible with the event type.

24429: Removing topics returns the number of topics removed

TopicControl#removeTopics now returns a Promise bearing a TopicRemovalResult. The result's property removedCount contains the number of non-reference topics that were removed as a result of the operation.

26120: Improve availability of JavaScript browser bundles outside of NPM

The diffusion-worker JavaScript bundle is now available in the on-premise installation in addition to the npm based installation.

Licensing

21921: The license file has been changed from licence.lic to license.lic

The Diffusion license filename has been changed from licence.lic to license.lic. The old filename will still be accepted if no file with the new name is found.

Server

23003: The CPU used by a mostly idle server has been reduced

In this release, server multiplexer threads use less CPU when they are not busy. This reduces the CPU load required to run the server when there are few updates.

Topic Views

23936: Topic view scalar path directives now apply to time series JSON topics

The topic view scalar path directive may now be used with time series topics that have a JSON event type as well as normal JSON topics. When used with time series topics the scalar value is always taken from the latest time series event.

24737: Topic views no longer update reference topic metrics and throttling if the derived value hasn't changed

In previous releases, a reference topic's metrics were updated whenever the source value changed, even if the reference topic's value was unchanged (so no value was published). For topic views with a throttle clause, the throttlers for each reference topic could be triggered, releasing an unchanged value that was later discarded. This could unnecessarily delay publication of a later, changed value.

From this release, a reference topic's metrics, and any associated throttler, are only updated if the reference topic value changes. This removes redundant processing and improves performance.

Topics

22702: TimeSeries topics can be updated with a user supplied timestamp

A variation of append has been added to the TimeSeries feature which accepts a timestamp. This allows the user to control the timestamp metadata of the TimeSeries topic event.

Deprecations in 6.6.0

.NET Client

22877: The IStream methods of the SendRequestToFilter IFilteredRequestCallback are deprecated

In previous releases, IFilteredRequestCallback.OnError() was called if a SendRequestToFilterAsync operation failed. In addition, the failure was also notified via an exception thrown by the task returned by SendRequestToFilterAsync. IFilteredRequestCallback.OnClose() was never called.

From this release, IFilteredRequestCallback.OnError() is no longer called. Applications should rely on the task returned by SendRequestToFilterAsync to detect errors. Both OnError() and OnClose() methods have been deprecated. In a future release, these methods will be removed from IFilteredRequestCallback and it will be changed not to inherit from IStream.

24845: SERVER_CLOSING close reason is deprecated

The SERVER_CLOSING CloseReason in ClientControl is deprecated and will be removed in a future release. This close reason is not emitted by the server.

Apple Client

24842: Deprecation of SERVER_CLOSING close reason

The SERVER_CLOSING CloseReason in ClientControl is deprecated and will be removed in a future release. This close reason is not emitted by the server.

C Client

24843: Deprecation of SERVER_CLOSING close reason

The SERVER_CLOSING CloseReason in ClientControl is deprecated and will be removed in a future release. This close reason is not emitted by the server.

Java & Android Client

24818: SERVER_CLOSING CloseReason in ClientControl is deprecated

The SERVER_CLOSING CloseReason in ClientControl is deprecated and will be removed in a future release. This close reason is not emitted by the server.

JavaScript Client

22879: The onError and onClose methods of the sendRequestToFilter FilteredResponseHandler are deprecated

In previous releases, FilteredRequestCallback.onError() was called if a sendRequestToFilter operation failed. In addition, the failure was also notified via exceptional completion of the sendRequestToFilter result. FilteredRequestCallback.onClose() was never called.

From this release, FilteredRequestCallback.onError() is no longer called. Applications should rely on the Result returned by sendRequestToFilter to detect errors. Both onError() and onClose() methods have been deprecated. In a future release, these methods will be removed from FilteredRequestCallback.

Removals in 6.6.0

Adapters

23392: The Push Notification Bridge has been removed

The Push Notification Bridge has been removed in this release.

The bridge packaged in previous releases can be used against this release. Please contact Push Technology if you are using the bridge, or have requirements to integrate Diffusion with push notification servers.

C Client

23114: session_set_maximum_outbound_queue_size removed

Removes session_set_maximum_outbound_queue_size from the API. If you require a different outbound queue size for your session, use diffusion_session_factory_maximum_queue_size.

23259: Deprecated Items Removed

Topic details related structures and functions have been removed.

In SVC_NOTIFY_SUBSCRIPTION_REQUEST_T structure, TOPIC_DETAILS (topic_details) has been replaced with TOPIC_SPECIFICATION (topic_specification).

In TOPIC_MESSAGE_T structure, TOPIC_DETAILS_T (details) has been replaced with TOPIC_SPECIFICATION_T (specification).

In SESSION_T structure, _topic_details_refs has been removed.

The file topic_details_types.h has been removed.
The enumeration TOPIC_TYPE_T is now in file topic_specification.h

24570: Session wills removed

Session wills have now been removed from the C client. If you require this functionality, please now consider using session-based automatic topic removal.

Client

20517: TopicEventListener/TopicEventStream removed

The deprecated TopicEventListener and TopicEventStream interfaces have been removed from all client SDKs and support for older clients using them has been removed from the server.

23017: TopicUpdateControl feature removed

The deprecated TopicUpdateControl feature has been removed from all client SDKs and support for it in older clients connecting to the server has also been removed.

Configuration

24258: The ClientServiceConfig compressionThreshold setting has been removed

The ClientServiceConfig compressionThreshold setting has been removed. This setting was no longer used and was deprecated in Diffusion 6.5.

Federation

23255: Support for fan-out connections from Diffusion versions before 6.2 removed

The server no longer supports fan-out connections from servers before version 6.2.

Security

20814: Legacy AuthenticationHandler interface removed

The deprecated legacy AuthenticationHandler interface has been removed from all client SDKs and support for older clients using it has been removed from the server.

Server

13044: Server support for SessionDetails services removed

Support for the SessionDetails-based API features available via version 5.9 clients has now been removed from the server. Version 5.9 clients will no longer be able to use the getSessionDetails and SessionDetailsListener capabilities with a version 6.6 server.

23116: Legacy AddTopic services removed from Server

A number of addTopic methods which used TopicDetails were removed from all clients in release 6.2. Support for these methods has now been removed which means that pre 6.2 clients will no longer be able to use these methods against a 6.6 server. This also means that clients using EOL releases (specifically 5.9) will no longer be able to add topics at all in a 6.6 server.

23140: getTopicDetails service removed from server

The deprecated getTopicDetails service was removed from all clients in version 6.4. Support for the service has now been removed from the server which means that pre-6.4 clients will no longer be able to use this feature.

Topics

19347: Legacy Fetch feature removed

The deprecated legacy fetch mechanism has now been removed from the Topics feature in all client SDKs and from the server (meaning that older clients cannot use it against a 6.6 server). Supporting interfaces (such as Content) have also been removed.

23016: Slave topics removed

The deprecated slave topic type has now been removed from all client SDKs and all support for it has been removed from the server. Older clients will no longer be able to create slave topics. Slave topics stored in persistence files will not be restored.

Fixes in 6.6.0

.NET Client

24471: Time series value stream event datatype compatibility problem

Value streams are compatible with parent types. For example, a IValueStream<IJSON> can deserialize values from a String topic type.

IValueStream<IEvent<IJSON>> did not propagate the values for time series topics with the String event type. This has now been fixed.

24909: Sessions can reconnect even if explicitly closed by another session

An issue has been resolved where the server allowed clients to reconnect during the reconnection timeout, even if they had been explicitly closed by another session. This would only occur if session replication was enabled.

Apple Client

24201: Incorrect handling of regular expression quotes in topic selector expressions

Split-path pattern and full-path pattern topic selectors allow the use of regular expression quotes (\Q, \E). In previous releases, the quotes were handled incorrectly: any backslash (\) character between the quotes was interpreted as the end of the literal section.

This bug has been fixed in this release. Now everything between \Q and \E is correctly treated as literal characters.

24473: Time series value stream event datatype compatibility problem

Value streams are now compatible with parent types. For example, a
PTDiffusionJSONValueStream can now deserialize values from a PTDiffusionString topic type.

PTDiffusionJSONTimeSeriesEventValueStream did not propagate the values for time series topics with the PTDiffusionString event type. This has now been fixed.

24658: HTTP Headers content checked with case-sensitive values

Addresses an issue where the Apple client fails to establish a session due case-sensitive mismatches on HTTP headers. This has now been resolved.

26248: Inconsistent HTTP Header format

An issue has been fixed regarding the HTTP header format in the Apple Client that certain proxy tools were unable to correctly validate

C Client

24202: Incorrect handling of regular expression quotes in topic selector expressions

Split-path pattern and full-path pattern topic selectors allow the use of regular expression quotes (\Q, \E). In previous releases, the quotes were handled incorrectly: any backslash (\) character between the quotes was interpreted as the end of the literal section.

This bug has been fixed in this release. Now everything between \Q and \E is correctly treated as literal characters.

24305: Selectors with descendant qualifiers not evaluated

Addresses an issue where selector strings, containing descendant qualifiers, were not evaluated in the C client. This has now been resolved.

24472: Time series value stream event datatype compatibility problem

Value streams are now compatible with parent types.

For example, a value stream of JSON datatype can deserialize values from a topic of String datatype.

Value streams for Time Series events of JSON datatype now propagate values for time series events of the String datatype.

Configuration

26028: Configured inbound thread pool sizes were ignored

Since Diffusion 6.5.0, the configured size of inbound thread pools has been ignored. Instead, a default value (5 threads) was used. This bug has been rectified in this release so the configured size will be used once again.

Console

23817: Can't type negative values in console topic editor

The topic value editor in the Diffusion management console did not accept negative values for int64 and double topics. This has been fixed.

23865: Sessions tabs defaults to metrics, even if not permissioned

The Diffusion management console presented an error message to users who browsed to the Sessions tab without the required permissions to view the default (session metrics) tab. The console will now show the sessions browser in this case.

23968: Disallow creating time series topics with DONT_RETAIN_VALUE

The Diffusion management console allowed users to configure new time series topics with the DONT_RETAIN_VALUE topic property, which is not allowed for time series topics and would be ignored. This property is no longer listed for time series topics.

24505: Open sessions tab shows wrong session as having been closed

In the Diffusion management console, after using the open sessions browser to close a session, the table entry for a different session would be removed. The correct entry is now removed in all cases.

24775: Console users can see add/modify metric collector buttons without required permissions

The Diffusion management console displayed buttons and forms for adding and modifying metric collectors, even if the user only had viewing permissions. Attempting to use these buttons would cause an error. These buttons are now only displayed for users with the appropriate permissions.

26094: Logging out on Security/Auth tab with changes breaks console

The Diffusion management console could display an unrecoverable error if a user attempted to log out while they had unsaved settings on the Authentication or Security tabs. Logging out will now automatically discard unsaved changes.

26127: Console breaks when changing topic view with insert clauses

The Diffusion management console topic view editor failed to correctly handle topic view insert clauses (added in 6.6.0 preview 1). Topic views insert clauses are no longer parsed incorrectly as path mappings.

Java & Android Client

23956: Flow control can stall update streams under high load

In previous releases, a bug could cause a Java client using update streams to never recover from flow control. This caused updates to stall. The bug has been fixed in this release.

24092: Incorrect handling of regular expression quotes in topic selector expressions

Split-path pattern and full-path pattern topic selectors allow the use of regular expression quotes (\Q, \E). In previous releases, the quotes were handled incorrectly: any backslash (\) character between the quotes was interpreted as the end of the literal section.

This bug has been fixed in this release. Now everything between \Q and \E is correctly treated as literal characters.

24399: Time series value stream event datatype compatibility problem

Value streams are compatible with parent types. For example, a ValueStream<JSON> can deserialize values from a String topic type.

ValueStream<Event<JSON>> did not propagate the values for time series topics with the String event type. This has now been fixed.

JavaScript Client

23385: Client should cascade to XHR protocol by default

Previously, the default options for connecting to the Diffusion server with the JavaScript client only included the WEBSOCKET transport. When run within a browser context, the default transports now include a fallback to XHR polling.

23895: Documented incorrect ordering of arguments to PartialJSON.with()

The order of arguments of PartialJSON.with() is now documented correctly

23896: Cannot use numbers/int64/float in JSON constraints

Previously int64 data was serialized inconsistently and for 32 bit integers and wrongly for negative 64 bit integers. This caused partial JSON constraints to fail for integer values. It also prevented negative 64 bit integers from being passed between clients. Integers are now serialized correctly and in line with other clients. Partial JSON constraints can be used with the Int64 datatype.

24095: Error when client attempts to establish an insecure connection from a secure (https) origin

Error "Failed to connect TypeError: Cannot read property 'close' of undefined" could occur when attempting to establish an insecure connection from a secure (https) origin. This problem has now been resolved.

24203: Incorrect handling of regular expression quotes in topic selector expressions

Split-path pattern and full-path pattern topic selectors allow the use of regular expression quotes (\Q, \E). In previous releases, the quotes were handled incorrectly: any backslash (\) character between the quotes was interpreted as the end of the literal section.

This bug has been fixed in this release. Now everything between \Q and \E is correctly treated as literal characters.

24474: Time series value stream event datatype compatibility problem

Value streams are compatible with parent types. For example, a ValueStream<JSON> can deserialize values from a String topic type.

ValueStream<Event<JSON>> did not propagate the values for time series topics with the String event type. This has now been fixed.

24510: Buffer implementation not supported outside of NodeJS

Previously, there was no access to the Buffer API when running the client in a browser context. The Buffer API is now available through diffusion.buffer.

24589: Request-response message handler calling responder.reject throws unexpected exception and prevents further handling

Calling responder.reject in request-response messaging caused an unexpected exception and prevented further handling. This has been fixed.

24598: Cannot pass true to responder.respond()

The responder in request-response messaging did not allow boolean values to be passed as a response. Booleans are now detected as valid JSON data.
An optional datatype can be passed as a second argument to responder.respond(). This has now been documented and is reflected in the TypeScript API.

24699: Failure to parse IPv6 host when creating new session

When creating a new session, the client failed to correctly extract the port number when IPv6 was used to specify the host. Now, a host string of the format [1234::1a2b:cd34:f56a:be89]:8080 will be interpreted correctly.

24797: messes.setRequestStream and timeseries.append should take topic types and plain JavaScript type constructors for datatypes

The implementation of setRequestStream allows topic types and plain JavaScript type constructors for datatypes. This is now correctly reflected in the public TypeScript API .

24820: RecordModel.get should allow obtaining a record by key only

The implementation of RecordModel.get could take a single key argument, but the documentation and the TypeScript API did not reflect this. This has now been corrected.

24887: First call of UpdateStream.set returns a TopicCreationResult

The first call to UpdateStream.set returns a TopicCreationResult. This was not reflected in the API. This has now been rectified to the TypeScript definitions and added to the documentation.

24938: QueryResult.merge throws exception: "Error: TypeError: latestByOriginalSequence[k].sequence.toNumber is not a function"

Time series QueryResult.merge() threw an exception when the two query results had events in common. Now, merge works as expected.

24945: diffusion-worker.js is missing from bundle

The diffusion-worker.js bundle was not included in the distribution package. It has now been added.

24948: Homepage link points to old version of the client

The homepage link in the client package now points to the correct version.

MQTT

24829: Server logs NullPointer exception if websocket close frame received before MQTT connect packet

In 6.6.0 preview 2, if a client opened a websocket connection with Sec-WebSocket-Protocol set to mqtt, then sent a close frame before the MQTT connect packet before the MQTT connect packet, the server would log a null pointer exception before closing the connection. The problem has been fixed in this release: the server will close the connection without logging.

Replication

26081: IllegalArgumentException when adding a topic

A concurrency bug introduced in Diffusion 6.5.0 could leave stale topics in the topic tree. One side-effect is that a subsequent attempt to add a topic could fail with an IllegalArgumentException. The bug has been fixed in this release.

Security

24035: Failures to process locked security resources

The API allows security roles and system authentication principals to be locked by a particular principal. Due to bugs in previous releases, processing of locked resources could fail:

A server could not boot against a Security.store or SystemAuthentication.store configuration file containing one or more lines that modify a locked resource. The work around was to remove the offending lines.

In a cluster, a modification to a locked resource was not properly replicated. Instead, an error message would be logged by all servers receiving the replicated update.

Both bugs have been fixed in this release. Locked security roles and system authentication principals now function as designed.

Server

24697: Time series topic stored byte metric not reset on sequence gap

Time series topic metrics were not reset when a sequence gap was detected. This affected topic metrics for these topics. A sequence gap may occur when a clustered topic is updated while offline with file persistence activated.

24715: StackOverflowError when calculating differences between values

In previous releases, when the rate and complexity of difference calculations caused the adaptive delta streaming algorithm to reduce the difference quality, it was possible for difference calculations to fail abnormally with a StackOverflowException. The bug has been fixed in this release.

24763: Sessions can reconnect even if explicitly closed by another session

A issue has been resolved where the server allowed clients to reconnect during the reconnection timeout even if they had been explicitly closed by another session. This would only occur if session replication was enabled.

24834: Server deadlock during start up

In previous releases, the server could potentially fail to start, logging a PUSH-000773 message indicating the server was making slow progress executing DefaultPicoContainer.addOrderedComponentAdapter. This problem has been resolved in this release.

25985: Hang in MpscBlockingConsumerArrayQueue.spinWaitForUnblock

Due to a bug introduced in release 6.6 preview 1, under very rare circumstances it was possible for the server to stall, with a thread spinning forever in MpscBlockingConsumerArrayQueue.spinWaitForUnblock. The bug has been fixed in this release.

26125: NullPointerException thrown by ServerMultiplexerStateImpl#enableReplication

A NullPointerException which was logged when recovering a time series topic from the replication log after restoring it from file persistence has been fixed.

26255: ArrayOutOfBoundsException in persistence compactor

An issue has been fixed where persistence compaction could fail when a time series topic contained only orphaned edits.

System Monitoring/Statistics

24735: The cumulative delta bytes metric is no longer increased if successive values are the same

The delta bytes metric accumulates the size of each delta update representing the difference between successive values. In previous releases, if a topic was updated with its current value the delta bytes metric was incorrectly increased by the number of bytes in the value. If the value hasn't changed, the server doesn't publish an update.

The bug has been corrected in this release. Now, if a topic is updated with its current value, only the value bytes metric is increased. Consequently, if a topic is updated with a series of identical values, the delta compression ratio will trend towards 100%, rather than 0%.

25983: diagnostics.js does not support Java 11

The diagnostics script has been updated so that it can be run under both versions 8 and 11 of Java.

Topic Views

24032: Topic view publishes new value to old topic before removing it if scalar mapping is used in a simple (non-derived) view mapping

A topic view specification that contained a scalar path mapping but no other JSON type value mappings could result in a new value on the source topic being published to the reference topic before the removal of the reference topic due to a change to the indicated scalar value.
This problem has now been resolved.

24415: Topic view insert produces corrupt output when the source topic is a simple value

In the initial preview release, use of topic view insert could potentially corrupt data in the reference topic if the source topic contained just a scalar value. This problem has now been fixed.

24580: Delayed topic views do not work with time series topics

There was a problem with topic views that use the 'delay' option if they selected a time series source topic. No resulting reference topic was ever published and an attempt to fetch the reference topic caused a ClassCastException.
This problem has now been resolved.

24727: Incorrect removal of reference topics due to topic view re-evaluation

A single topic view can map more than one source topic to the same reference topic path. In this situation, the first selected source topic (in tree path order) creates and updates the reference topic, but others are ignored.

A problem with the processing of removals in this situation meant that the removal of any source topic that could generate the reference topic would result in removal of the reference topic. The topic would quickly get created again, but this would cause unexpected unsubscriptions and resubscriptions for clients.

This problem has now been resolved.

26096: Failure to process topic view if an expand clause has a path pointer that does not match the expanded structure

The topic view expand clause has an optional a path pointer that can be used to extract an alternative part of the expanded value to use in the derived path in place of the natural key.

In previous releases, processing of topic views with expand clauses could fail in arbitrary ways if the path pointer did not match the expanded value. The bug has been fixed in this release.

26206: Missing or incorrect values in reference topics when topic view evaluations are chained

In some cases where a topic view generated reference topics that were then evaluated as source topics for another topic view, other reference topics generated by the original view were not generated as expected (either not created or with invalid values).
This problem has now been resolved.

Topics

24231: Potential failure to process time series updates

In previous releases, a rapid sequence of updates to time series topics could fail to be processed correctly, leading to data corruption. This bug has been fixed in this release.

24234: Automatic topic removal policy using "when this session closes" - topic can be removed while session still exists

If a topic had the 'when this session closes' REMOVAL clause and the creating session failed over to another server in the cluster, the topic would be removed. This was incorrect: the topic should remain as long as the original session does.

A similar problem could also occur if the creating session had its session properties changed in any way.

These problems have now been resolved.

24413: Topic updates with the same value do not prevent automatic topic removal

In previous releases, updating a topic repeatedly to the same value may not have prevented it being removed by automatic topic removal using the 'no updates for' clause. This has now been resolved.

Known Issues

Console

26133: Console shows fractional users connected

The concurrent clients and client connections graphs on the console can display fractional users.

Topic Views

26130: Reference topics retained by 'preserve topics' are not persisted across server instances or cluster

A new 'preserve topics' clause has been introduced to topic views in release 6.6. This clause means that reference topics created by a view (that have a path dependent upon the source topic value) are retained until the source topic is removed or the topic view is removed. Though this is true in the context of a single server instance, it is not the case if the server is restarted as all such topics created during the previous server instance will be lost. It is also not the case if a new server enters a cluster as the new server will only have reference topics generated from the point in time where it joined the cluster and will not reflect reference topics previously created within other cluster peers.

This issue occurs because reference topics are not persisted, either to file or across the cluster.

This issue will be resolved in a future release.