Just a second...

Topic views

A topic view is a dynamic way to map one part of a server's topic tree to another. This enables the server to transform topic data before sending it to clients. Remote topic views enable mapping of topics between separate servers.

Source topics and reference topics

A topic view maps topics from one part of the topic tree to another. It takes a set of existing source topics and dynamically creates a set of reference topics, based on a declarative topic view specification.

The reference topics can be a simple mirror, but a topic view can transform the topics in various ways, for example by changing their values or properties, or throttling the rate of publication.

You can use a remote topic view to map part of the topic tree on one Diffusion™ the Diffusion server to another.

You can also use topic view inserts to merge data from topics other than the source topics into the reference topics.

Topic views are useful to create reference topics with the exact data required by clients, minimising the need to process data on the client. For example, you can transform a source topic with a large, complex JSON value into a set of simpler reference topics. Clients can then subscribe only to the individual topics they need.

Reference topics are read-only. They cannot be updated, nor can they be created or removed directly. Otherwise, they behave just like standard topics. Each reference topic has the same topic type as its source topic.

Client sessions can subscribe to a reference topic, and can fetch its current value if it has one.

The topic specification of a reference topic is derived from the topic specification of the source topics. A reference topic has the same topic type as its source topic.

The source topics of a topic view are defined by a topic selector.

If a source topic is removed, reference topics that are derived from it will automatically be removed. If a topic is added that matches the source topic selector of a topic view, a corresponding reference topic will be created. Removing a topic view will remove all of its reference topics.

There is no hard-coded limit to the number of topic views you can create.

One-to-one mapping and expansion

In most topic views, the mapping is one-to-one: each source topic generates a single reference topic.

Starting in Diffusion 6.4, a topic view can use an expand value directive to create multiple reference topics from the values within a JSON source topic.

This is useful if you have a complex JSON topic with more information than each client requires. You can use an expand value directive to unpack the topic into multiple reference topics, and each client can subscribe to the relevant topics.

Note: Topic view expansion is a powerful feature which should be used with care to avoid performance issues. If the topic structure of the generated reference topics changes frequently, it can lead to high server load.

See below for full details of how to use an expand value directive.

Topic view specifications

The following is a simple topic view specification that mirrors all topics below the path a to reference topics below the path b.

map ?a// to b/<path(1)>

A topic view with this specification will map a source topic at the path a/x/y/z to a reference topic at the path b/x/y/z. The specification is simple, so the reference topic will exactly mirror the source topic.

A general topic view specification contains these clauses:

  • The source topic clause identifies the source topics.
  • The path mapping clause determines how reference topic paths are derived from the source topic paths.
  • The insert clause (optional) can be used to specify values or partial values from topics that are not part of the specified source topics; the specified data is inserted into reference topics (must be JSON).
  • The topic property mapping clause determines how reference topic properties are derived from source topic properties.
  • The topic value mapping clause (optional) determines how reference topic values are derived from source topic values.
  • The optional throttle clause (optional) constrains the rate at which each reference topic is updated when its source topic is updated.

For readability, clauses can be separated by line endings. Lines prefaced with # are treated as comments.

An example topic view specification is:

map ?A//
from Server1
to <path(1)>
as <value(/foo)>
# Join 2 topics
insert Topic2 at /T2
insert Topic3 at /T3
throttle to 1 update every minute
		

Source topic clause

The source topic clause begins with the map keyword and is followed by a topic selector.

The following is an example of a source topic clause:

map ?foo/bar//

This matches the topics "foo", "bar" and all of their descendants. See Topic selectors for the full topic selector syntax, which enables you to specify part of the topic tree using powerful regular expressions.

Note: If any of the topic names contain spaces, wrap the selector in quotation marks. For example, instead of map ?Results/Home games//, use map "?Results/Home games//".

When evaluating a topic view, the following topics are ignored even if they match the selector:

  • Topics created through the Publisher API
  • ROUTING topics

Both slave topics and reference topics are valid source topics.

Chaining of topic views is supported; that is, a reference topic created by one topic view can be the source topic of another topic view. A reference topic can be the master topic of a slave topic, or the source topic of a routing topic subscription.

Note that slave topics can do some of the functions of topic views, but topic views are preferable because of the more sophisticated mapping options.

Path mapping clause

The path of a reference topic is derived from the source topic according to the topic view path mapping. Both the path of the source topic and its value can be used to determine the path of the reference topic.

A path mapping clause begins with the to keyword and is followed by a path mapping template.

A path mapping template is a topic path with embedded directives. Directives are evaluated when creating the topic reference and substituted into the topic path.

Directives are delimited by angle brackets (<, >) and consist of the name of the directive and a list of parameters. The parameter list is comma-separated and surrounded by parentheses (( , )).

The following example adds a simple path mapping clause to the previous source topic clause:

map ?foo/bar// to /foo/bar/All/<path(3)>

The path mapping clause is the part after the to keyword. path is a directive, with "3" as its only parameter. The clause will make reference topics for all the topics under foo/bar at foo/bar/All in the topic tree.

There are three types of path mapping directive:

Source path directives

A source path directive matches part of the source path as defined by the source topic clause. This enables you to derive the path of the reference topics from all or part of the path of the source topics.

The syntax is <path(start, number)>:
  • start is the index of part of the source path (where the top level of the topic tree is indexed as 0)
  • number is the number of parts to include (optional; if not specified, the selection will extend to the end of the source path)

For example, if you have a source path a/b/c/d:

  • The source path directive <path(0, 2)> is mapped to the reference topic path a/b. The start parameter of 0 means start at index 0, which is the topic a, and the number parameter of 2 means to take two parts of the tree in total.
  • The source path directive <path(1)> is mapped to the reference topic path b/c/d. This time the start parameter is 1, which corresponds to the topic b. Because there is no number parameter, the match includes all the topics to the end of the path.
Source value directives

Source value directives enable you to derive the path of the reference topics from the values within a JSON source topic.

Source value directives are only applied to JSON source topics; if the path mapping contains a source value directive, non-JSON topics matching the source topic selector are ignored.

Source value directives use the keyword scalar and are parameterized by a single JSON pointer that extracts a scalar value from the source value.

A scalar value is a string, a number, true, false, or null; that is, anything other than an array or a object.

If the JSON pointer does not refer to a scalar value in the source value, no reference topic will be created. This includes the cases where the JSON pointer refers to an array or an object), or when no part of the source value is selected.

Deriving the reference topic paths from part of the source topic value effectively creates a secondary index on the value. For source value directives to work efficiently, the selected scalar values should be relatively stable. If an update to the source topic changes the selected scalar value, the corresponding reference topic will be removed and a new reference topic will be created.

For example, given a source value of:
{
	"account" : "1234",
	"balance" : { "amount" : 12.57, "currency" : "USD" }
}
and a source value directive:

currency/<scalar(/balance/currency)>/account/<scalar(/account)>

the resulting reference topic path will be:

currency/USD/account/1234

If the extracted value is a string, it is copied literally to the reference topic path. A value that contains path separators (/) will create a reference topic path with more levels than the path mapping template.

An extracted value of null will be copied to the reference topic path as the string "null".

Expand value directives

Expand value directives enable you to create multiple reference topics from a single JSON source topic.

Expand value directives are only applied to JSON source topics; if the path mapping contains an expand value directive, non-JSON topics matching the source topic selector are ignored.

Expand value directives use the keyword expand and take one or two JSON pointers as parameters:

<expand(sourcePointer, pathPointer)>

The sourcePointer parameter is a JSON pointer used to specify an array or object to expand.

Every direct child of the indicated source value is expanded into a new reference topic. If no pointer is specified, the value is expanded from the root.

The optional pathPointer parameter can be used to name the topic path. A JSON pointer is used to specify a scalar value (that is, not an array or object) within the expanded value. The value is used to derive the reference topic path.

For example, suppose this is the content of a JSON topic called allCars:
{
	"cars": [
		{ "reg":"HY58XPA", "type":"Ford", "model":"Sierra" },
		{ "reg":"PY59GCA", "type":"Fiat", "model":"Panda"},
		{ "reg":"VA63ABC", "type":"Ford", "model":"Ka"}
		]
} 

The following topic view specification:

map allCars to cars/<expand(/cars, /reg)>

results in these reference topic paths:

	cars/HY58XPA
	cars/PY59GCA
	cars/VA63ABC
The value of cars/HY58XPA is:
	{
		"reg":"HY58XPA",
		"type":"Ford",
		"model":"Sierra"
	}

If the second pointer is not specified or no scalar value is found for the pointer, the path fragment is taken from the key (if the child value is an object) or the index (if the child value is an array).

In the example above, if you used the topic view
map allCars to cars/<expand(/cars)>
, the topic path is taken from the index of the current array element resulting in topics:
cars/0
cars/1
cars/2

Expand directives can be nested (that is there can be more than one expand directive in a path mapping). In this case, a second expand directive will use the value from the previous expand as its source (root) value and not the value of the source topic. This also applies to scalar directives that follow an expand directive.

Suppose the previous array example is extended so that each car can have multiple drivers:
{
	"cars": [
		{ "reg":     "HY58XPA", 
		"drivers": [{"name" : "Bill"}, {"name" : "Fred"}]
		},
		{ "reg":     "PY59GCA",
		"drivers": [{"name" : "Jane"}, {"name" : "Fred"}]
		},
		{ "reg":     "VA63ABC", 
		"drivers": [{"name" : "Tom"}, {"name" : "John"}]
		}
	]
} 

This topic view uses nested expand directives to expand both levels of the array hierarchy:

map allCars to cars/<expand(/cars, /reg)>/drivers/<expand(/drivers, /name)>

resulting in these reference topics:
cars/HY58XPA/drivers/Bill
cars/HY58XPA/drivers/Fred
cars/PY59GCA/drivers/Jane
cars/PY59GCA/drivers/Fred
cars/VA63ABC/drivers/Tom
cars/VA63ABC/drivers/John
The second expand directive takes the /drivers values from the previous expand, so the value of each topic is {"name" : "Bill"}, {"name" : "Fred"}, {"name" : "Jane"} and so on.

If expansion causes more than one mapping to the same topic path, only the first encountered will be created and updated.

Insert clause

The optional insert clause can be used to insert data from topics other than the selected source topics, enabling you to merge data from mutiple topics together.

You can insert data from JSON or scalar (integer/string) topics. The reference topics receiving the data must be JSON topics.

You can either insert the whole value of a topic, or part of a JSON value.

You can specify which part of a JSON topic to insert by using a JSON pointer. You can also use a JSON pointer to specify where to insert within the reference topic.

The insert clause has the form:

insert path_specification key source_key at insertion_key default constant

for example:

map ?Some_Source_Topics/ to Mapped_Topics/<path(1)> insert Some_Other_Topic at /Some_JSON_Pointer

path_specification

This specifies the path of the non-source topic from which data is to be obtained and merged with the current input data value (in the simplest case, this is the value from the selected source topic; this can depend on the other clauses).

The path specification uses similar syntax to the path mapping clause.

The specification can contain:

  • Constants
  • <path()> directives operating on the path of the selected source topic
  • <scalar()> directives operating on the current input data value

For example:

Topic/<path(0, 2)>/<scalar(/foo)>

specifies insertion from a topic whose path is Topic/ followed by the first three elements of the source topic path, followed by the value at the key /foo in the current input data value.

Note that this enables you to select additional nonsource topics to merge based on the structure and content of the source topics.

source_key

This optionally specifies the key (a JSON pointer) of an item within the topic indicated by path_specification. If not specified, the whole of the data value of the selected topic will be inserted.

insertion_key

This specifies a JSON pointer indicating the location of the insertion in the current input data value.

Typically this is an object key indicating the key of the value in the data.

If the data already has an item with the same key it is overwritten. Otherwise, a new item is added to the parent indicated by the specified key. If the parent does not exist, the insertion does not occur and a warning is logged.

This can also indicate an entry in an array. If an index key is provided, the existing entry at the specified index is replaced. Use the special - character to append to an array. For example, to append to the end of an array at MyArray use /MyArray/- as the insertion key.

If the resolved key indicates a scalar item, no insertion will take place.

default constant

Usually if insertion fails, nothing happens. The default keyword can be used to override this behavior by providing a default value to insert.

Insertion can fail if:
  • the topic specified to insert from (using path_specification) cannot be found
  • the topic is not a JSON/scalar topic
  • the specified key does not exist

If default is used, if insertion fails, the specified constant value is inserted as a scalar value at the insertion point.

The current input data value is typically the value from the selected source topic.

However, there are some situations where this is not the case:
  • if the path mapping includes one or more expand directives, each of the expanded data values is treated as the current input data value in turn. If a path mapping is used to expand an array of five elements, then the insert clause is executed 5 times, once for each element.
  • if the insert was preceded by an as<value(key)> directive, the current input data is the data indicated by the key.
  • if the insert was preceded by another insert clause, then the current input data is the output from that clause; this enables input clauses to be chained.

Example insert clauses:

map Topic1 to Topic2 insert AnotherTopic at /key default "unknown"

Map Topic1 to Topic2, and the data within AnotherTopic is inserted into it at the key named other.

map ?Topics/ to Mapped/<path(1)> insert AnotherTopic at /other

Like the previous example, but all of the topics under the path Topics will be selected and mapped to topics with the same name under the path Mapped. Every selected topic will have the value of AnotherTopic inserted into it (assuming they are JSON objects) unless AnotherTopic does not exist, in which case no insertions would take place.

map ?Topics/ to Mapped/<path(1)> insert Others/<path(1)> at /other

This uses the more powerful path mapping capabilities of topic inserts. In this case, each selected topic has an insertion from a topic with the same topic under the path Others. For example, Topics/A/B would generate a reference topic at path Mapped/A/B which has the value of Others/A/B inserted at the key other.

map ?Topics/ to Mapped/<path(1)> insert Others/<scalar(/foo)> at /other

Similar to the previous example, but in this case the path of the insertion topic is derived from a value within the selected source topic. So if topic Topics/A/B has a value of "bar" at key "foo", the topic selected to insert from would be Others/bar.

map ?Topics/ to Mapped/<path(1)> insert Others/<path(1)> key /foo at /other

All previous examples have shown the insertion of the whole value of another topic. Here the key keyword is used to select a specific item foo within the insertion topic value. If the insertion topic does not have the value with key foo, then a reference topic will be created but no insertion will occur.

When expand directives are used, the insert will occur for every output from the expansion. For example:

map Topic1 to Expanded/<expand()> insert AnotherTopic at /other

If we assume that the content of Topic1 is an array of objects then each array element would be expanded to produce a new topic at path Expanded/0, Expanded/1 and so on, and each resulting reference topic will have the value from AnotherTopic inserted at the key /other.

Insert clauses can be chained together:

map Topic1 to Topic2 insert AnotherTopic at /other insert YetAnotherTopic at /yetAnother

In the above example values from two different topics are inserted into the data to produce the reference topic.

And finally, the insert clause can be used along with as <value()> clauses, for example:

map Topic1 to Topic2 insert AnotherTopic at /foo/bar as <value(/foo)>

In this example the data from AnotherTopic is inserted at the key foo/bar, then the full value of foo is projected.

Topic property mapping clause

The topic properties of a reference topic are derived from the source topic. Some topic properties can be changed using the optional topic property mapping clause.

A topic property mapping clause begins with the keywords with properties and consists of a comma-separated list of topic property keys and values, each separated by a colon. For example, the following topic view specification maps all topics below the path a to reference topics below the path b, and disables both conflation and compression for the reference topics.

map ?a// to b/<path(1)> with properties CONFLATION:off, COMPRESSION:false

The following table describes the behavior for each topic property.

Table 1. Reference topic property mapping
Source topic property Reference topic spec default Set by topic property mapping? Notes
COMPRESSION Copied from source topic specification Yes  
CONFLATION Copied from source topic specification Yes  
DONT_RETAIN_VALUE Copied from source topic specification Yes  
OWNER Not set No  
PERSISTENT Not set No Reference topics are not persisted. Topic views are persisted, so a reference topic will be recreated on server restart if its source is persistent.
PUBLISH_VALUES_ONLY Copied from source topic specification Yes  
REMOVAL Not set No Reference topics cannot be removed directly.
SCHEMA Copied from source topic specification No A RECORD_V2 reference topic has the same schema as its source topic.
SLAVE_MASTER_TOPIC Not set No If a reference topic has a slave topic as its source topic, it indirectly references the slave's master topic.
TIDY_ON_UNSUBSCRIBE Copied from source topic specification Yes  
TIME_SERIES_EVENT_VALUE_TYPE Copied from source topic specification No A TIME_SERIES reference topic has the same value type as its source topic.
TIME_SERIES_RETAINED_RANGE Copied from source topic specification Yes, with restrictions A topic property mapping cannot increase the time series retained range by overriding the TIME_SERIES_RETAINED_RANGE property. The retained range of a reference time series topic will be constrained to be no greater than that of its source topic.
TIME_SERIES_SUBSCRIPTION_RANGE Copied from source topic specification Yes  
VALIDATE_VALUES Not set No A reference topic reflects updates to its source topic. It cannot reject updates.

Topic value mapping

The value of a reference topic is derived from the source topic according to the topic view value mapping. By default, a reference topic's value is the same as its source topic.

For JSON source topics, the optional topic value mapping clause can be used to extract part of the source value. A topic value mapping begins with the keyword as and is followed by a value directive.

A value directive is delimited by angle brackets (<, >), and consists of the value keywords and a single JSON pointer parameter. The JSON pointer selects the part of the source value to copy. For example, given a source value of:
{
	"account" : "1234",
	"balance" : { "amount" : 12.57, "currency" : "USD" }
}
and the value mapping clause as <value(/balance)>, the reference topic value will be:
{
	"amount" : 12.57,
	"currency" : "USD"
}
Topic value mappings are often used with path value mappings. For example:
map ?accounts// to balances/<scalar(/account)> as <value(/balance)>

Throttle clause

The optional throttle clause can be used to constrain the rate at which a reference topic is updated when its source topic is updated. The primary application of a throttle clause is to restrict the number of updates sent to reference topic subscribers, reducing network utilization or the processing each subscriber must do. Throttling also restricts the rate at which client sessions can observe changes to reference topic values using the fetch API.

The throttle clause has the form:

throttle to X updates every period

where X is a positive integer, and period is a positive integer followed by a time unit: seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most twice every five seconds:

map ?a// to b/<path(1)> throttle to 2 updates every 5 seconds

To improve readability, the throttling clause allows 1 update as an alternative to 1 updates, and every second as an alternative to every 1 seconds (as well as for other time units).

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most once every hour:

map ?a// to b/<path(1)> throttle to 1 update every hour

The throttle clause is only applied when a source topic is updated more frequently than the configured rate. If a source topic is updated less frequently, updates are passed on unconstrained. If the rate is exceeded, a reference topic will not be updated again until the configured period has expired. At this time, the reference topic will be updated based on the source topic updates that happened in the interim, and a single value will be published. Thus, a throttle clause provides topic-scoped conflation.

The throttle clause is ignored for time series topics because time series updates do not support efficient conflation. Updates to source time series topics are passed on immediately to the corresponding reference topics, regardless of any throttle clause.

Delayed topic views

You can use the optional delay clause to create a delayed topic view. This means that changes to the source topics are reflected in the reference topics after a fixed delay of your choice.

The delay period can be from one second to many days.

A delayed topic view can be useful if you provide real-time data as part of a paid service, and want to provide a delayed feed as a preview or for a lower service tier.

The maximum delay possible is constrained by the disk space available on the Diffusion server to store the delayed updates (in a delay_store subdirectory of the persistence directory). The amount of space required will vary depending on the details of your application, such as how many source topics are included in the view and how frequently they update.

The delay does not just apply to topic values: topic additions and removals are also delayed, so the reference topics will reflect the structure of the source topics as they were at the earlier point in time.

The delay clause has the form delay by duration where duration is a positive integer followed by a time unit which is one of seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, with a five minute update delay.

map ?a// to b/<path(1)> delay by 5 minutes

Views with delay clauses initially create reference topics in an unpublished state. The topics are published once the delay time has passed.

Note that if you restart the Diffusion server, the delay time will need to pass before any reference topics are published. Delayed events are not persisted.

An unpublished reference topic prevents a lower priority topic view from creating a reference topic with the same path.

Client sessions with the rights to read the source topic can browse unpublished topics using a fetch request.

Remote topic views

A remote topic view maps its source topics from a remote server.

This is indicated with a from clause:

map ?a/ from Server1 to b/<path(1)>

The above example maps the topics under topic node a on a remote server to topic node b on the server where the topic view was created.

Server1 is the name of a remote server definition, created using any client API.

If a topic view is created with a remote server definition that does not exist, the topic view will remain inactive until the remote server definition is created and a connection is established to the remote server.

Any number of topic views can share a remote server definition and map topics from the same server.

If a remote server definition is removed when one or more topic views refer to it, reference topics created by those views will be removed and the views will become inactive. If the remote server is recreated, the views will reactivate. This ensures that network connections are only in place for remote servers when there are topic views that require them.

When a connection to a remote server connection is lost, the reference topics of all topic views that refer to it will be removed.

Normal topic view precedence rules are applied to remote views. If two topic views try to create reference topics on the same path, the view that was created first takes precedence, whether it is local or remote.

If a remote view has no working server connection, it will not prevent a younger local view creating a reference topic on the same path; but if the remote view then establishes a connection, precedence will be applied, and the reference topic from the local view is replaced by one mapped from the remote view.

In a Diffusion cluster, both topic views and remote server definitions are automatically distributed across the cluster. Each member of the cluster will automatically connect to the same remote server and produce the same reference topics from the remote topic views.

All of the other topic view capabilities can be applied to a remote topic view: for example, a remote topic view can have an expand value directive or a throttle clause.

Quoting and white space

Topic selectors and path mapping templates can be quoted or unquoted. They are quoted using the single quote mark. To include whitespace, single quotes or literal opening angle brackets they must be quoted. In quoted selectors and templates single quotes, literal opening angle brackets and backslashes must be escaped with a single backslash. In templates the opening angle bracket should be unescaped when beginning a directive. Characters in unquoted selectors and templates can't be escaped.

Any whitespace can be used to separate keywords, statements and clauses.

Dealing with topic path conflicts

If you create a topic view which tries to make a reference topic with the same path as an existing topic, the result is a topic path conflict.

Reference topics have a lower priority than normal topics created through the API, including replicas of normal topics created by topic replication or fan-out. A reference topic will only be created if no topic or reference topic is already bound to its derived topic path.

Topic views have a precedence based on order of creation. If two topic views define mappings to the same topic path, the earliest-created topic view will create a reference topic. If a topic view is updated, it retains its original precedence.

Topic view persistence and replication

Reference topics are neither replicated nor persisted. They are created and removed based on their source topics. However, topic views are replicated and persisted. A server that restarts will restore topic views during recovery. Each topic view will then create reference topics based on the source topics that have been recovered.

The server records all changes to topic views in a persistent store.

Topic views are restored if the server is started. If a server belongs to a cluster, topic views will be replicated to each server in the cluster. Topic views are evaluated locally within a server. Replicated topic views that select non-replicated source topics can create different reference topics on each server in the cluster.

Access control

The following access control restrictions are applied:
  • To list the topic views, a session needs the READ_TOPIC_VIEWS global permission.
  • To create, replace, or remove a topic view, a session needs the MODIFY_TOPIC_VIEWS global permission and SELECT_TOPIC permission for the path prefix of the source topic selector.
  • Each topic view records the principal and security roles of the session that created it as the topic view security context. When a topic view is evaluated, this security context is used to constrain the creation of reference topics. A reference topic will only be created if the security context has READ_TOPIC permission for the source topic path, and MODIFY_TOPIC permission for the reference topic path. The topic view security context is copied from the creating session at the time the topic view is created or replaced, and is persisted with the topic view. The topic view security context is not updated if the roles associated with the session are changed.