Diffusion 6.4 Release Notes
6.4.1 (9 Jan 2020)
Bugs cleared at release 6.4.1
Console: Topic view topic selector assistant: HTML escaping of topic hints is incorrect (21564)
In 6.4.0, when adding or editing a topic view, the explanatory hints in the topic selector assistant form were incorrectly escaped more than once, leading to garbled text. This is corrected in this release; the hints are now displayed correctly.
Console: Load average graph flatlines by default (21593)
The Load Average graph shown by default in the Diffusion management console was misconfigured and constantly showed 0 load. The default configuration has been fixed. You may need to restore the default layout in order for the fix to be applied.
Console: Fetching an only-child leaf topic causes infinite fetch loop (21626)
Viewing certain topic tree structures caused the Diffusion console to repeatedly fetch data from the server. This resulted in an increase in overall server load, more noticeably when using topic views with value mapping. This issue has now been fixed.
Topics: Errors in topic REMOVAL property ignored (21559)
Some errors in a topic REMOVAL property were ignored by the Diffusion server instead of being reported. For example, if the property was set to "when time after 'Thu, 05 Dec 2019 12:45:20 GMT' and subscriptionsZ < 1 for 10s'" the server would simply ignore the second clause (which is invalid due to using "subscriptionsZ") instead of reporting an error. This has now been fixed.
Topics: Topic views expand directive causes excessive topic churn when JSON object keys inserted or removed (21577)
The initial release of topic views expansion processed objects in key order. Insertions or removals of keys within an object could cause topics to be created and deleted unnecessarily (topic churn). Expansion now considers keys and derived paths to minimize topic churn.
6.4.0 (5 Dec 2019)
Features introduced at release 6.4.0
Client: Partial updates for JSON topics (17945)
The new partial update feature can be used to apply a JSON Patch (RFC 6902) to a JSON topic. This means that the updating client does not need to provide the full value. The JSON Patch standard supports advanced operations such as moving and copying key/value pairs, as well as testing a value before applying the patch. The dispatch of partial updates is supported by all the client SDKs.
Client: Fetch requests can limit the results returned from topic tree branches with long paths (21149)
A fetch option has been added that limits the number of topic results returned for each branch having paths with greater than a certain number of parts. See FetchRequest.limitDeepBranches().
Console: Topic view editing assistance (20549)
Extensive editing assistance has been added to the console's topic views tab. This includes syntactic highlighting, contextual help, and context-specific "assistant" dialogs for the various parts of a topic view specification.
Fan-out: Fan-out support for the topic REMOVAL property (18706)
In previous releases, automatic topic removal did not take into account sessions and subscribers at secondary fan-out servers, and was therefore of limited use in fan-out configurations. From this release, all sessions and subscriptions at secondary fan-out servers are taken into account when evaluating topic removal conditions.
Fan-out: Fan-out readiness start condition (21250)
In this release, each connector in etc/Connectors.xml can be configured with a list of conditions that must be satisfied before the connector is started. A single type of condition is supported, "fanout-ready", which is satisfied when a named fan-out link has connected to its primary server and initially synchronized all matching topics from the primary server. This can be used to delay opening of the connector port until there is information to serve to sessions.
Installation: Improved installer (6492)
The installer has been rewritten for this release. The new installer is based on the third party izpack framework. The installation options have been simplified. In previous releases, both the installer and a separate product distribution jar file were required. In this release, there is a single installer file that should be run using Java, For example: java -jar diffusion-installer-6.4.0.jar
Added closeSession method to Session that returns a Result. The result resolves once the session has been closed.
Diffusion.connect now accepts a httpProxyAgent option when run on Node.js. This allows the client to establish connections through a proxy server.
Server: Officially support Java 11, and OpenJDK derived distributions (20542)
From this release, the Diffusion server and Java client SDK are supported on a wider range of Java implementations. Specifically any Java Virtual Machine (JVM) that meets the following requirements: 1. The JVM either passes the Java TCK or is an official AdoptOpenJDK build; 2. the JVM version is 8u131-b11 (GA) or a later minor GA release of JDK 8, or 11.0.3 (GA) or a later minor GA release of JDK 11; 3. the JVM is HotSpot based; 4. the target platform is Linux, macOS, or Windows.
Server: The COMPRESSION topic property provides control over the compression level (21475)
From this release, the COMPRESSION topic property enables different compression levels to be set to trade off the CPU cost against the compression level achieved. The property now accepts the new values "off", "low", "medium", and "high", in addition to the previous values "true" (equivalent to "medium") and "false" (equivalent to "off"). The default compression level has been reduced to "low".
Topics: Topic delivery priority (15924)
A PRIORITY topic property has been added that can be used to set the delivery priority of topic updates. The supported values are "high", "default" and "low". The delivery priority affects the order of topic updates sent to a subscribed client session. When there are multiple topic updates for topics with different priorities in a session's outbound queue, updates for high priority topics will be delivered first, followed by updates for default priority topics, followed by updates for low priority topics. Topic subscription/unsubscription notifications are also delivered according to the topic delivery priority. Using different delivery priorities is most beneficial when there is a large backlog of queued updates to deliver to a client session. On lightly loaded systems, updates typically remain in the outbound queue for a few milliseconds and so there is a lower chance of topic updates being reordered based on their priority. The backlog will be larger if the topic update rate is higher; the server or the client are more heavily loaded; the client session becomes temporarily disconnected; or if there is poor network connectivity between the server and the client.
Topics: Topic views: new 'expand' directive (20655)
At this release, a major new capability has been added to topic views. There is a new 'expand' directive which allows the content of a JSON topic to be expanded into many different reference topics. For example, each element of an array from a source topic can be expanded to produce one reference topic per array entry, each being updated automatically by updates to the source topic data. See the documentation for full details of this feature.
Changes made to existing features at release 6.4.0
.NET Client: NewTopicSpecification convenience method added to Diffusion factory (19688)
A NewTopicSpecification factory method has been added.
.NET Client: Deprecated topic-selector interfaces, builders, and methods (20372)
Topic selectors and the methods that take them as arguments are now deprecated. Methods that accept a topic selector expression as string should be used instead.
.NET Client: Topics.GetTopicDetails and TopicAddFailReason.USER_CODE_ERROR removed (20699)
The deprecated ITopics.GetTopicDetails methods have now been removed along with all ITopicDetails interfaces (which were only retained to support these methods). The TopicAddFailReason.USER_CODE_ERROR has also been removed.
Adapters: The JMS adapter is now only supported as a standalone client (20872)
In preparation for the removal of the Publisher API (deprecated in this release), the server edition of the JMS adapter has been removed from the product.
Android Client: Support for Android API 29 (21532)
From this release, the Android client is additionally compatible with Android API 29.
Apple Client: WebSocket channels support for compress encoding (13541)
The Apple client now supports the transfer of compressed messages over WebSocket connections.
Apple Client: PTDiffusionFetchRequest.first/last accept 0 as a value (20433)
PTDiffusionFetchRequest.first and PTDiffusionFetchRequest.last can now accept a 0 value. Fetching zero topics which match a selector can be used to detect the existence of matching topics by using PTDiffusionFetchResult.hasMore.
C Client: System principals hash map now has SYSTEM_PRINCIPAL_T as its entry value (20903)
The system principals hash map (from a SYSTEM_AUTHENTICATION_STORE_T) now has SYSTEM_PRINCIPAL_T pointer values as opposed to previously SET_T.
Client: MissingTopicNotification.cancel deprecated (20950)
The MissingTopicNotification.cancel method has been deprecated at this release. This method is only useful when using the deprecated legacy fetch mechanism and will be removed when that mechanism is removed.
Console: Server configuration for console-exposed log files location (19611)
The log file monitored in the Diffusion management console can now be changed using the new 'console-monitored-log' setting in the Diffusion logging configuration. This makes it easier to use the console with custom logging configurations.
Console: Enable/disable functionality based on permissions (20449)
Some areas of the console, including the Security view, Topic Views and Metrics Collectors tabs, are now available in read-only mode to users with the appropriate permissions. Users should now only be presented with actions that they are permitted to attempt (topic tree permissions notwithstanding).
Console: Console topics tab improvements (20633)
The topics tab in the Diffusion web console has received a number of new features and usability improvements. You can now update topic values directly from the console. JSON and binary topic values are formatted for readability. The form to add a topic now informs you if the target path is already in use and allows for multiple topics. The console now asks for confirmation when subscribing to or deleting a topic. Many minor visual issues have also been corrected.
Console: Console topic browser performance significantly improved with large topic trees (20729)
Previously, using the console's topic browser to explore large topic trees could cause the console to disconnect or become unresponsive. Following improvements in 6.4, the topic tree browser should consistently perform well with topic trees of any size.
Console: Console display of locked roles and principals (20861)
The Security tab in the console now displays information on locked roles and principals, and will prevent users without the lock from attempting to edit them.
Console: "Add and continue" button to create multiple topics quickly in the console (21166)
When creating topics using the Diffusion web console, you no longer have to reopen the topic adder to create more than one topic.
Environment: docker-diffusion: diffusion home no longer depends on version (20761)
The docker-diffusion build no longer uses a version-specific path for the Diffusion home directory. Diffusion is installed to /opt/diffusion in the Docker container's filesystem.
Installation: The ec2 tools examples have been removed (21379)
Diffusion now includes systemd unit files. Upstart init scripts have been removed.
Java & Android Client: Java client now supports Server Name Indication extension to SSL/TLS (20214)
The Java client now supports the SNI extension for SSL/TLS handshakes. This is always enabled, and uses the host name that the client is connecting to.
Java & Android Client: Topics.getTopicDetails and TopicAddFailReason.USER_CODE_ERROR removed (20514)
The deprecated Topics.getTopicDetails methods have now been removed along with all TopicDetails interfaces (which were only retained to support these methods). The TopicAddFailReason.USER_CODE_ERROR has also been removed.
Java & Android Client: Consistent behaviour of CompletableFuture exceptions (20616)
In previous releases, some methods that returned CompletableFuture could complete exceptionally with an ExecutionException or an API specific exception, rather than with a CompletionException. Also, the javadoc for some older releases indicated that methods could complete exceptionally with an ExecutionException when they would actually complete exceptionally with a CompletionException. These inconsistencies have now been resolved. All such methods will now complete exceptionally with a CompletionException as now indicated by the javadoc.
The Result type is now a type alias for Promise.
Two new CloseReason enum values have been added. PROTOCOL_VERSION_MISMATCH indicates that the sever responded to a connection request with a protocol version number different to the client's protocol version. LICENSE_EXCEEDED indicates that a connection was refused because the license limit has been reached.
Logging: Log4j is now the default logging implementation (20544)
The default logging implementation is now Log4j 2. The Diffusion logging implementation (which was the default system in previous versions) is deprecated, together with most of the configuration settings in etc/Logs.xml, and will be removed in a future release.
Product Release: The docker-diffusion image is now derived from AdoptOpenJDK (20150)
The docker-diffusion image published to Docker Hub is now based on the AdoptOpenJDK OpenJDK 8 distribution. It has been updated to jdk8u232-b09.
Publisher API: Publisher API Deprecated (20553)
The Publisher API is deprecated at this release and will be removed in a future release. All aspects of topic and client handling can be performed using the Diffusion Client API. Some remaining aspects of the server-side API, such as configuration and HTTP request handling, will be retained and will in future be known as the 'Server API'.
Replication: Routing of authentication handler requests across Diffusion server clusters (20547)
Previous releases needed an authentication handler connected to every server in a cluster. At this release, authentication requests will be routed across the cluster to a server that has a suitably named handler. Handlers can be connected to more than one server for greater efficiency, but only one is required. It is the responsibility of the application to ensure that all handlers with the same name have the same behavior. Also, at this release, it is a requirement for on-premise users that the configuration of authentication handlers (in Server.xml) is the same for every server in a cluster.
Replication: Missing topic notifications are now routed across Diffusion server clusters (20893)
Missing topic notifications are now forwarded to handlers registered on other servers in the same cluster if a suitable handler is not found locally. If the server producing the event is a fanout secondary, it will still propagate the request to the primary, but the handling server will not.
Security: Principals and roles can now be locked by statements in the corresponding stores (20596)
The server now supports more granular control over the principals and roles. Previously, the MODIFY_SECURITY permission was sufficient to modify any principal or role in the security and system authentication stores. The new behavior is to optionally associate a principal or role with a principal that can edit it. The syntax for creating a principal is: `add principal 'name' 'password' locked by 'lock_principal'`. The syntax for locking a role is: `set role 'role_name' locked by 'lock_principal'` The locking principal still requires the MODIFY_SECURITY permission to modify the store. The corresponding script builders and queries to the stores have been updated to support this functionality.
Server: Improved delta calculation performance when inputs have many differences (20818)
When there is a large number of differences found in the input values, the difference engine backs off to balance the amount of CPU required against the size of the resulting delta. Improvements to the back-off algorithm have significantly reduced the CPU cost of comparing values that have a lot of differences, without further increasing the size of the delta.
Server: The Connector MBean setQueueDefinition() operation is deprecated (21232)
The connector MBean setQueueDefinition() operation has no effect. It is deprecated and will be removed from a later version of the product.
System Monitoring/Statistics: The Diffusion Publisher and JMX Adapter have been removed (20543)
Previous releases required that a special "Diffusion" publisher was deployed to support console functionality. The publisher exposed various JMX MBeans below a Diffusion/MBean topic branch. From this release, the console no longer relies on the JMX adapter. The JMX adapter and the Diffusion publisher have been removed.
System Monitoring/Statistics: Publish product version and license information to Prometheus (20902)
The server now exports information about the Diffusion product version and license to Prometheus as diffusion_release and diffusion_license metrics.
System Monitoring/Statistics: diffusion_peak_sessions Prometheus metric now a gauge (21231)
The diffusion_peak_session metric exported to Prometheus has been changed from a counter to a gauge because it is reasonable to access the metric without using rate().
Topics: Improve evaluation of automatic topic removal policies for topics restored from persistence (19901)
Evaluation of automatic topic removal rules for topics restored from persistence is now delayed until all topics have been restored. This ensures that all restored topics that match the removal selector will be removed.
Topics: Slave topics deprecated (20282)
Slave topics have been deprecated at this release. Use topic views instead. The effect of a slave topic can be duplicated by creating a topic view that maps a single topic to a single reference topic. Topic views add far more powerful functionality such as mapping whole branches of the topic tree.
Topics: Topic Views : Support for value mapping in JSON time series topics (20509)
Topic views now apply value mapping clauses to JSON time series topics. Previously JSON time series topics would have been ignored by views that specified value mapping clauses.
Topics: Quoting and escaping of characters in topic view expressions has been simplified (21059)
Quoting and escaping of special characters in topic view expressions has been simplified. In previous releases, escape sequences had several limitations. They could only be used in mapping clauses that were wrapped in quotes. Only a few characters could be escaped and the characters that could be escaped varied for different parts of the expression. They could not be used to escape special characters in JSON Pointer directive parameters. In this release, these limitations have been removed. The escape sequence \x can be used to literally insert any character x, with one exception: \/ cannot be used in path fragments. As a side effect of these changes, the root JSON pointer (i.e. the empty string) can now be used in <scalar> and <value> directives, as well as the new <expand> directive.
Bugs cleared at release 6.4.0
C Client: Update stream memory allocation issues (20894)
Addresses an issue where incorrectly-sized memory allocations were enacted when creating an update stream.
C Client: SECURITY_STORE_ROLE_T not present in public API (20931)
Addresses an issue where the SECURITY_STORE_ROLE_T struct was not present in the public API (as part of security-control.h). This has now been addressed.
Configuration: SecurityConfig does not prevent duplicate control authentication handlers (20765)
When configuring an embedded Diffusion server using the SecurityConfig class, multiple control authentication handlers could have the same name, leading to later issues. This has now been resolved and authentication handler configuration is properly validated.
Console: Console log issues handling Server.log file roll over (19681)
The console's log viewer would stop updating and sometimes display errors when the underlying log file was truncated, e.g. due to log rollover. The console now automatically follows log rollovers in tailing mode, and allows the user to fetch the latest log in browse mode.
Console: Selecting uncreated topics in the dashboard produces confusing results (20815)
Previous versions of the console could show outdated values in the console dashboard when tracking topics that did not exist. This is now fixed, and a warning is provided when you select a topic which does not currently exist.
Console: Console sends unescaped topic names in selectors, causing disconnection (21208)
Topics with names containing special topic selector characters could cause visual problems and disconnections in the console. This issue has now been fixed.
Console: Add sesson metric collector button looks disabled when name missing but still works (21407)
Previously the console sometimes allowed users to add metrics collectors without a name. This is no longer possible.
Handlers registered with the server are closed if their callback methods throw an exception. When this happens, they will not continue to receive notifications from the server. However, in previous versions, the server would continue to attempt to send notifications to the handler. The server is now informed of the handler close and stops sending notifications to it.
The client now reliably allows unsetting a topic with a primitive datatype by passing null or undefined to topicUpdate.set()
Licensing: Issues with cluster licensing view of cluster (20158)
The cluster maintains a dynamic view of the number of connections the cluster is licensed for. Earlier implementations have been subject to race conditions that caused different servers in the cluster to report inconsistent limits. Changes have been made to ensure eventual consistency.
Licensing: Soft limit breached warning at every connection (21123)
A log message is generated when the soft connection limit is breached. Previously it was generated for each connection above the soft limit. Now a message is only logged when the soft limit is first exceeded.
Replication: Replica streams do not recover after reconnection (21303)
Previous releases had a bug where topic replication did not recover correctly following the loss and reconnection of a connection between two servers. This would prevent replication of topic updates for some of the topics on some of the servers. The bug has been fixed in this release.
Replication: Cluster fails to form correctly following split brain (21370)
In previous releases, there were situations in which the topic replication service was not informed of Hazelcast partitions being moved following resolution of a split-brain condition. Consequently, some topic operations would persistently fail and report that "the cluster is re-partitioning". In this release, additional recovery code has been added to detect and resolve this problem.
Replication: A transient stale value can be published during recovery (21375)
When a server joins or leaves a cluster using topic replication, the server responsible for coordinating updates to a topic can change. When this happens, the new server notifies its peers and provides the current value of the topic. In previous releases, a server performing this "recovery" process could incorrectly re-publish an older value before publishing the latest value, resulting in some subscribers incorrectly getting the out-of-date value before being sent the correct value. This problem has been fixed in this release.
Server: Sessions using the HTTP polling transport can fail if there is a large backlog of messages (20929)
Due to a bug, sending a large backlog of messages to a session using the HTTP polling transport could occasionally fail, resulting in the session being closed with an IO_EXCEPTION exception. The bug has been fixed in this release.
Server: NullPointerException looking up GeoIP data (20994)
A NullPointerException could be thrown when looking up location and locale information for certain IP addresses. This can happen for IP addresses that are associated with the EU rather than a country in the EU. These IP addresses are now identified and the root locale is provided.
Server: Using JMX to restart a connector causes an unnecessary warning in the log (21244)
In previous releases, if JMX was used to stop and restart a connector, the server erroneously logged a PUSH-000195 "InstanceAlreadyExistsException" message to the logs. From this release, the message will no longer be logged.
Server: When the GeoIP database is enabled, the server incorrectly assigns country name to $Country (21348)
In previous releases, when the GeoIP database was enabled, the server incorrectly assigned a country name to the $Country session property. In this release, the server has been corrected to use the country ISO code, as documented.
Topics: Scalar topic view directives can evaluate to an illegal path (20897)
The use of scalar directives in a topic view could result in invalid paths. For example, if the scalar directive matched an empty string, a path with an empty segment could be returned. When attempting to add a reference topic at this path, an exception would be thrown. This has been addressed by filtering invalid topic paths and preventing the attempt to create reference topics for them.
Console: Console shows fractional users connected (21562)
The concurrent clients and client connections graphs on the console can display fractional users.