Just a second...

Topic views

A topic view is a dynamic way to map one part of a server's topic tree to another. This enables the server to transform topic data before sending it to clients. Remote topic views enable mapping of topics between separate servers.

Source topics and reference topics

A topic view maps topics from one part of the topic tree to another. It takes a set of existing source topics and dynamically creates a set of reference topics, based on a declarative topic view specification.

The reference topics can be a simple mirror, but a topic view can transform the topics in various ways, for example by changing their values or properties, or throttling the rate of publication.

You can use a remote topic view to map part of the topic tree on one Diffusion™ the Diffusion server to another.

You can also use topic view inserts to merge data from topics other than the source topics into the reference topics.

Topic views are useful to create reference topics with the exact data required by clients, minimising the need to process data on the client. For example, you can transform a source topic with a large, complex JSON value into a set of simpler reference topics. Clients can then subscribe only to the individual topics they need.

Reference topics are read-only. They cannot be updated, nor can they be created or removed directly. Otherwise, they behave just like standard topics. Each reference topic has the same topic type as its source topic.

Client sessions can subscribe to a reference topic, and can fetch its current value if it has one.

The topic specification of a reference topic is derived from the topic specification of the source topics. A reference topic has the same topic type as its source topic.

The source topics of a topic view are defined by a topic selector.

If a source topic is removed, reference topics that are derived from it will automatically be removed. If a topic is added that matches the source topic selector of a topic view, a corresponding reference topic will be created only if any transformations that follow can be performed. Removing a topic view will remove all of its reference topics.

There is no hard-coded limit to the number of topic views you can create.

One-to-one mapping and expansion

In most topic views, the mapping is one-to-one: each source topic generates a single reference topic.

Starting in Diffusion 6.4, a topic view can use an expand value directive to create multiple reference topics from the values within a JSON source topic.

This is useful if you have a complex JSON topic with more information than each client requires. You can use an expand value directive to unpack the topic into multiple reference topics, and each client can subscribe to the relevant topics.

Note: Topic view expansion is a powerful feature which should be used with care to avoid performance issues. If the topic structure of the generated reference topics changes frequently, it can lead to high server load.

See below for full details of how to use an expand value directive.

Topic view specifications

The following is a simple topic view specification that mirrors all topics below the path a to reference topics below the path b.
map ?a// to b/<path(1)>

A topic view with this specification will map a source topic at the path a/x/y/z to a reference topic at the path b/x/y/z. The specification is simple, so the reference topic will exactly mirror the source topic.

A topic view specification comprises three main parts:
  • The mapping part which specifies the source topics to map from and the mappings to target reference topics.
  • Optional transformations which transform the topic value in some way.
  • Optional options which specify other changes that the view may apply.
Mapping comprises:
  • The source topic clause identifying the source topics that the view can apply to.
  • The optional from clause which may identify a remote server that hosts the source topics.
  • The path mapping clause which determines how reference topic paths are derived from the source topic paths, and when expanding to more than one reference topic, from where the values are obtained.
Transformations can be:
  • patch transformation(s) specifying that a JSON patch is applied to the reference topic value.
  • process transformations that allow conditional processing and/or calculations to be applied to the reference topic value.
  • insert transformation(s) specifying that values from other topics are inserted into the reference topic value.
Options can be:
  • The topic property mapping clause determines how reference topic properties are derived from source topic properties.
  • The value mapping clause determines how reference topic values are derived from source topic or expanded values.
  • The throttle clause constrains the rate at which each reference topic is updated when its source topic is updated.
  • The delay by clause causes a change to a view's source topic to be delayed by a fixed time before it is reflected in reference topics.
  • The separator clause can define a replacement path separator for values extracted using the scalar or expand directives.
  • The type clause can specify that the reference topic that is created is of a different topic type from the selected source topic.

For readability, clauses can be separated by line endings. Lines prefaced with # are treated as comments.

An example topic view specification is:
map ?A//
from Server1
to <path(1)>
# Join 2 topics
insert Topic2 at /T2
insert Topic3 at /T3
throttle to 1 update every minute
as <value(/foo)>

Source topic clause

The source topic clause begins with the map keyword and is followed by a topic selector.

These topic selectors follow the same parsing rules as other topic selectors.

The following is an example of a source topic clause:
map ?foo/bar//
This matches the topics "foo", "bar" and all of their descendants. See Topic selectors for the full topic selector syntax, which enables you to specify part of the topic tree using powerful regular expressions.
Note: If any of the topic names contain spaces, wrap the selector in quotation marks. For example, instead of map ?Results/Home games//, use map "?Results/Home games//".

When evaluating a topic view, all topics in the topic tree that match the source topic selector are considered (excluding ROUTING topics).

However, if a view specification uses some feature that can only be applied to JSON topics then only JSON topics and, in some cases, JSON time series topics will be selected.

Reference topics are valid source topics. In particular, chaining of topic views is supported; that is, a reference topic created by one topic view can be the source topic of another topic view.

Additionally, a reference topic can be the source topic of a routing topic subscription.

From clause

The 'from' clause optionally follows the source topic clause. It begins with the from keyword and is followed by a remote server name.

The name refers to a remote server created using the Remote Servers feature.

The presence of the clause indicates that the source topics will be selected from the specified server and not from the local server.

Further details regarding the processing of remote topic views are given below.

Path mapping clause

The paths of reference topics are derived from the source topic according to the path mapping clause.

The path mapping allows the source topic path and the value of the source topic to determine the path of the reference topic.

In addition the path mapping can include expand directives which allow objects and arrays in JSON source topic values to be expanded to produce many reference topics.

A path mapping clause begins with the to keyword and is followed by a path mapping template.

A path mapping template is a topic path with embedded directives. Directives are evaluated when creating the topic reference and substituted into the topic path. Directives are delimited by angle brackets (<, >) and consist of the name of the directive and a list of parameters. The parameter list is comma-separated and surrounded by parentheses ((, )).

The following path mapping directives are supported:

Source path directives

Source path directives extract a portion of the source path and are parameterized by the index of the start part of the source path and the number of parts to include. The number of parts parameter is optional – if it is missing, the selection extends to the end of the source path.

The syntax is <path(start, number)>, or <path(start)> when the number of parts parameter is omitted.

For example, given the source path a/b/c/d, the source path directive <path(1, 2)> is mapped to the reference topic path b/c, and the source path directive <path(2)> is mapped to the reference topic path c/d.

Source value directives

Source value directives are only applied to JSON source topics or TIME_SERIES source topics with JSON event type; if the path mapping contains a source value directive, topics with other topic types matching the source topic selector are ignored.

Source value directives use the keyword scalar and are parameterized by a single JSON pointer that extracts a scalar value from the source (or current) value. A scalar value is a string, a number, true, false, or null, that is, anything other than an array or a object. If the JSON pointer does not refer to a scalar value in the source (or current) value, no reference topic will be created. This includes cases where the JSON pointer refers to an array or an object, or when no part of the source value is selected.

Deriving the reference topic paths from part of the source topic value effectively creates a secondary index on the value. For source value directives to work efficiently, the selected scalar values should be relatively stable. If an update to the source topic changes the selected scalar value, the corresponding reference topic will be removed and a new reference topic will be created.

For example, suppose this is the content of a JSON topic called allCars:
{
    "cars": [
        { "reg":"HY58XPA", "type":"Ford", "model":"Sierra" },
        { "reg":"PY59GCA", "type":"Fiat", "model":"Panda"},
        { "reg":"VA63ABC", "type":"Ford", "model":"Ka"}
    ]
}
The following topic view specification: map allCars to cars/<expand(/cars, /reg)> results in these reference topic paths:
cars/HY58XPA
cars/PY59GCA
cars/VA63ABC
The value of cars/HY58XPA is:
{
    "reg":"HY58XPA",
    "type":"Ford",
    "model":"Sierra"
}

If the second pointer is not specified or no scalar value is found for the pointer, the path fragment is taken from the key (if the child value is an object) or the index (if the child value is an array).

In the example above, if you used the topic view map allCars to cars/<expand(/cars)>, the topic path is taken from the index of the current array element resulting in topics:
cars/0
cars/1
cars/2

Expand directives can be nested (that is there can be more than one expand directive in a path mapping). In this case, a second expand directive will use the value from the previous expand as its source (root) value and not the value of the source topic. This also applies to scalar directives that follow an expand directive.

Suppose the previous array example is extended so that each car can have multiple drivers:
{
    "cars": [
        { "reg":     "HY58XPA",
          "drivers": [{"name" : "Bill"}, {"name" : "Fred"}]
        },
        { "reg":     "PY59GCA",
          "drivers": [{"name" : "Jane"}, {"name" : "Fred"}]
        },
        { "reg":     "VA63ABC",
          "drivers": [{"name" : "Tom"}, {"name" : "John"}]
        }
    ]
}
This topic view uses nested expand directives to expand both levels of the array hierarchy: map allCars to cars/<expand(/cars, /reg)>/drivers/<expand(/drivers, /name)> resulting in these reference topics:
cars/HY58XPA/drivers/Bill
cars/HY58XPA/drivers/Fred
cars/PY59GCA/drivers/Jane
cars/PY59GCA/drivers/Fred
cars/VA63ABC/drivers/Tom
cars/VA63ABC/drivers/John

The second expand directive takes the /drivers values from the previous expand, so the value of each topic is {"name" : "Bill"}, {"name" : "Fred"}, {"name" : "Jane"} and so on.

If expansion causes more than one mapping to the same topic path, only the first encountered will be created and updated.

For example, given a source value of:
{
    "account" : "1234",
    "balance" : { "amount" : 12.57, "currency" : "USD" }
}

and the source value directive currency/<scalar(/balance/currency)>/account/<scalar(/account)>, the reference topic path will be currency/USD/account/1234.

If the extracted value is a string, it is copied literally to the reference topic path. A value that contains path separators (/) will create a reference topic path with more levels than the path mapping template. Use the separator directive to replace path separators with an alternative string.

An extracted value of null will be copied to the reference topic path as the string "null".

Transformations

Transformations are specified after the mapping and before any options. Transformations can only be applied to JSON topics.

Transformations are applied to the value extracted from the source topic in the order specified. There can be any number of transformations interspersed with one another and the value from one will be that which is input to the next. The only restriction is that all insert transformations must occur at the end, after any others.

A transformation is applied to the current value within a view processing chain, so if a transformation occurs after an expand then it will be applied to each expanded value.

Process transformations

Process transformations may be used to apply conditional processing to a value (optionally determining whether a reference topic is created) and/or change the value in some way (for example, by applying some calculation to a field within the value).

The format of a process transformation is:
process {statement}
Where the statement can be:
  • Operation(s).

    One or more operations separated by ';'.

  • A conditional statement.

    Comprising one or more conditions with operations to perform if they are satisfied.

For example, the following topic view specification could be used to write a field into the value of the reference topic:
map ?a// to b/<path(1)> process {set(/Name, 'John')};
The following example shows a simple conditional statement which would only generate reference topics if the value of field /Price was greater than 50:
map ?a// to b/<path(1)> process {if '/Price gt 50' continue};
And the following shows a more complex statement which would set a field according to the value of the input field /Price:
map ?a// to b/<path(1)> process {if '/Price lt 50' set(/Tier, 1) elseif '/Price gt 50' set(/Tier, 2)};
Process operations

The following operations are supported:

Table 1. Supported process operations
Operation Description
set(pointer,value)

Sets the field indicated by the JSON pointer to an absolute value. If the field does not exist it will be created. The value can be an integer (e.g. 123), a String (e.g. "XYZ"), or a boolean (e.g. true or false).

For example :
set(/Name, "John")
If a hierarchic pointer is specified, the parent object or array must exist.
set(pointer, calc calculation)

Sets the field indicated by the JSON pointer to a value which is the result of the specified calculation. If the field does not exist it will be created. The calculation can include fields within the input value. The calculation is specified as a quoted string. See below for a detailed description of calculations.

For example:
set(/DoubleValue, calc "/Value * 2")
If a hierarchic pointer is specified, the parent object or array must exist.
remove(pointer)

Removes the json item at the specified pointer. Unlike JSON patch, if the item does not exist the operation does not fail.

continue

This is a special operation that indicates that the topic view evaluation should continue with the value as it is. This is only for use with conditional statements as the default behavior of a conditional statement is not to proceed if no condition is satisifed.

Operations can be chained by separating them with a ';' as shown in the example below:
set(/Amount, calc "/Value * /Number"); remove(/Value); remove (/Number)

In this case the operations are all performed on the original value, creating a chain of deltas which are only applied to the original value at the end. If the 'set' operation fails no reference topic would be generated, however, processing will continue if the fields specified in the 'remove's are not present.

Process calculations

A calculation may be specified as the value of a set operation. A calculation is a simple arithmetic calculation upon integer fields. If applied to a non integer field the evaluation will not proceed. Floating point calculations are not supported.

Arithmetic operators supported are +, -, * and /.

Examples of calculations are:
set(/Value, calc "/Value * 2")
set(/Result, calc "/Value / 2")
set(/Bonus, calc "/Salary + 1000")
set(/Bonus, calc "/Salary + 1000 + /Age * 10")

Standard operator precedence is applied, so in the last example above we have (/Salary + 1000 + (/Age * 10)) not ((/Salary + 1000 + /Age) * 10). Brackets may be used to override this.

Process conditional statements

A conditional statement is made up of an if clause, optionally followed by one or more elseif clauses and an optional final else clause.

The if clause takes the form:
if condition operation(s)

Where the condition is a quoted string as described in detail below and the operation(s) is as described previously. If the condition is satisfied, the operations are applied to the value and the process is complete. If the condition is not satisfied, processing moves on to any elseif or else clauses that follow, but if there are none, the topic view evaluation does not proceed and no reference topic is created.

An elseif (else if) clause takes the form :
elseif condition operation(s)

If the condition is satisfied, the operations are applied to the value and the process is complete. If the condition is not satisfied, processing moves on to any elseif or else clauses that follow, but if there are none, the topic view evaluation does not proceed and no reference topic is created. Note that elseif can be abbreviated to elsf.

An else clause takes the form:
else operation(s)

And will only be reached if no previous if or elseif conditions were satisfied. If reached then the operations are applied to the value and the topic view evaluation proceeds. The continue operation may be used to proceed with an unchanged value.

Process conditions
A condition is of the form:
pointer operator [constant/pointer]

Where pointer is a JSON pointer, operator is a relational operator and constant is a string, integer, or boolean value.

For example:
/Age > 40
/Name = "Bill"
/Age > /RetirementAge
/Manager eq true

Operators allowed are:

Table 2. Allowed process condition operators
Operator Variant Description Supported JSON types
= eq Equals All
> gt Greater than Integer only
< lt Less than Integer only
!= ne Not equals All
>= ge Greater than or equal Integer only
<= le Less than or equal Integer only

Compound conditions are supported by means of boolean operators:

| or or

& or and

For example:
/Age = 50 or /Age > 80
/Age gt 50 & /Department eq "Accounts"
Normal boolean precedence applies but brackets can be used to control precedence. For example:
(/Age > 50 or /Department eq "Accounts") and /Band > 3
Boolean 'not' is also allowed :
not (/Age < 65 or /Retired eq false)

Patch transformations

Patch transformations indicate that a JSON patch is to be applied to the value.

The format of a patch transformation is:
patch 'patch string'

The patch string should be formatted according to the JSON Patch standard (see RFC 6902: JavaScript Object Notation (JSON) Patch).

Patches are a sequence of JSON Patch operations contained in an array. They are applied as an atomic update to the previous value if the resulting update is successfully calculated. The following patch will check the value at a specific key and update if the expected value is correct:
[{"op":"test", "path":"/price", "value" : 22}, {"op":"add", "path":"/price", "value": 23}]
The available operations are:
  • Add: {"op": "add", "path": "/a/b/c", "value": ["foo", "bar" ]}
  • Remove: {"op": "remove", "path": "/a/b/c"}
  • Replace: {"op": "replace", "path": "/a/b/c", "value": 43}
  • Move: {"op": "move", "from": "/a/b/c", "path": "/a/b/d"}
  • Copy: {"op": "copy", "from": "/a/b/c", "path": "/a/b/e"}
  • Test: {"op": "test", "path": "/a/b/c", "value":"foo"}

The test operation checks that the CBOR representation of the value of a topic is identical to the value provided in the patch after converting it to CBOR. If the value is represented differently as CBOR, commonly due to different key ordering, then the patch will return the index of the failed operation. e.g the values {"foo": "bar", "count": 43} and {"count": 43, "foo": "bar"} are unequal despite semantic equality due to the differences in a byte for byte comparison.

The following patch clause would add the 'price' field and remove the 'name' field from an input JSON object.
patch '[{"op":"add", "path":"/price", "value" : 22}, {"op":"remove", "path":"/name"}]'

Patches can only be applied to JSON arrays or objects and if they fail to apply, no resulting reference topic will be created by the view. If an update patch fails, any previously created reference topic would be removed.

Insert transformations

Insert transformations are used to insert a value from another topic into the current value.

The name of the topic to insert from can be specified in a similar way to the path mapping in that constants, path directives, and scalar directives (but not expand directives) may be used.

The value from the insertion topic (if found) is inserted into the 'current' value at a specified key position. The current value may be the source topic value, the value output from expand directives (in which case the insertion applies to each value), or the value from a previous transformation. Insertion topics may be JSON, STRING, INT64, DOUBLE.

If, when evaluating a topic view, the insertion topic is not found (or the specified key within it is not found), or it is of an unsupported topic type, an optional default value may be inserted, but if no default is specified then no insertion takes place and the value is passed to the derived reference topic unchanged.

The values of the insertion topics are only taken at the point when the source topic is evaluated against the topic view (i.e. when the source topic is updated). Changes to the value of the insertion topic that occur in the interim are not applied to the derived reference topics.

The format of an insert transformation is
insert path [key fromKey] at insertionKey
[default defaultValue]

The path is specified in exactly the same way as for the path mapping clause, except it may not contain expand directives. path directives operate on the path of the source topic, whereas scalar directives operate on the 'current' value as defined previously.

key is optional and can specify a fromKey which is a JSON pointer indicating the data within the insertion topic that is to be inserted. If no key is specified then the whole of the insertion topic value is inserted.

at specifies the insertionKey which is a JSON pointer indicating where to insert the insertion topic value. If a value already exists at the specified key then it is replaced otherwise it is inserted. Insertion can only occur if the parent of the key exists in the value and is of a compatible type. Array pointers may only be used to replace existing entries or append one greater than the last entry. The special pointer value /- may be used to append to the end of an existing array.

default is optional and may be used to specify a string defaultValue to be inserted if the insertion topic does not exist, it is of an unsupported topic type, or the specified key within it could not be found.

The following insert transformation would cause the whole value of the topic named AnyTopic to be inserted into the current value at key /T, assuming that the current value is an object.
insert AnyTopic at /T
The following insert transformation would cause the whole value of the topic named AnyTopic to be inserted into the current value at key /T/MyKey, assuming that an object with the key T exists in the current value.
insert AnyTopic at /T/MyKey
The following insert transformation would cause the whole value of the topic named AnyTopic to be appended to the array at the key T in the current value.
insert AnyTopic at /T/-
The following insert transformation would cause the value at the key name within the topic named AnyTopic to be appended to the array at the key T in the current value.
insert AnyTopic key /name at /T/-
In the above examples no insertion would take place if the insertion value was not found, but a default value can be specified to insert into the current value in this case.
insert AnyTopic at /T/- default "unknown"
The path of the topic to insert from can be built from parts of the source topic path and/or scalar values within the current value. For example:
insert AC/<path(1,1)>/<scalar(/myval)> at /T
Example insert clauses:
map Topic1 to Topic2 insert AnotherTopic at /key default "unknown"
Map Topic1 to Topic2, and the data within AnotherTopic is inserted into it at the key named other.
map ?Topics/ to Mapped/<path(1)> insert AnotherTopic at /other
Like the previous example, but all of the topics under the path Topics will be selected and mapped to topics with the same name under the path Mapped. Every selected topic will have the value of AnotherTopic inserted into it (assuming they are JSON objects) unless AnotherTopic does not exist, in which case no insertions would take place.
map ?Topics/ to Mapped/<path(1)> insert Others/<path(1)> at /other
This uses the more powerful path mapping capabilities of topic inserts. In this case, each selected topic has an insertion from a topic with the same topic under the path Others. For example, Topics/A/B would generate a reference topic at path Mapped/A/B which has the value of Others/A/B inserted at the key other.
map ?Topics/ to Mapped/<path(1)> insert Others/<scalar(/foo)> at /other
Similar to the previous example, but in this case the path of the insertion topic is derived from a value within the selected source topic. So if topic Topics/A/B has a value of "bar" at key "foo", the topic selected to insert from would be Others/bar.
map ?Topics/ to Mapped/<path(1)> insert Others/<path(1)> key /foo at /other
All previous examples have shown the insertion of the whole value of another topic. Here the key keyword is used to select a specific item foo within the insertion topic value. If the insertion topic does not have the value with key foo, then a reference topic will be created but no insertion will occur.
When expand directives are used, the insert will occur for every output from the expansion. For example:
map Topic1 to Expanded/<expand()> insert AnotherTopic at /other

If we assume that the content of Topic1 is an array of objects then each array element would be expanded to produce a new topic at path Expanded/0, Expanded/1 and so on, and each resulting reference topic will have the value from AnotherTopic inserted at the key /other.

Insert clauses can be chained together:
map Topic1 to Topic2 insert AnotherTopic at /other insert YetAnotherTopic at /yetAnother

In the above example values from two different topics are inserted into the data to produce the reference topic. And finally, the insert clause can be used along with as <value()> clauses, for example:

map Topic1 to Topic2 insert AnotherTopic at /foo/bar as <value(/foo)>
In this example the data from AnotherTopic is inserted at the key foo/bar, then the full value of foo is projected.

Options

Options are specified after the mapping and any transformations. Any number of options may be specified but any particular option may only be specified once.

Topic property mapping

The topic specification of a reference topic is derived from the topic specification of the source topics. A reference topic has the same topic type as its source topic.

The topic properties of a reference topic are derived from the source topic. Some topic properties can be tuned using the topic property mapping option. The following table describes the behavior for each topic property.

Table 3. Reference topic property mapping
Source topic property Reference topic specification default Cat be set by topic property mapping? Notes
COMPRESSION Copied from source topic specification Yes  
CONFLATION Copied from source topic specification Yes  
DONT_RETAIN_VALUE Copied from source topic specification Yes  
OWNER Not set No  
PERSISTENT Not set No Reference topics are not persisted. Topic views are persisted, so a reference topic will be recreated on server restart if its source is persistent.
PRIORITY Copied from source topic specification Yes  
PUBLISH_VALUES_ONLY Copied from source topic specification Yes  
REMOVAL Not set No Reference topics cannot be removed directly.
SCHEMA Copied from source topic specification No A RECORD_V2 reference topic has the same schema as its source topic.
TIDY_ON_UNSUBSCRIBE Copied from source topic specification Yes  
TIME_SERIES_EVENT_VALUE_TYPE Copied from source topic specification No A TIME_SERIES reference topic has the same value type as its source topic.
TIME_SERIES_RETAINED_RANGE Copied from source topic specification Yes, with restrictions A topic property mapping cannot increase the time series retained range by overriding the TIME_SERIES_RETAINED_RANGE property. The retained range of a reference time series topic will be constrained to be no greater than that of its source topic.
TIME_SERIES_SUBSCRIPTION_RANGE Copied from source topic specification Yes  
VALIDATE_VALUES Not set No A reference topic reflects updates to its source topic. It cannot reject updates.
A topic property option begins with the keywords {@code with properties} and consists of a comma-separated list of topic property keys and values, each separated by a colon. For example, the following topic view specification maps all topics below the path a to reference topics below the path b, and disables both conflation and compression for the reference topics.
map ?a// to b/<path(1)> with properties CONFLATION:off, COMPRESSION:false

Topic value mapping option

By default, a reference topic's value is a copy of the source topic value, or part of the source value produced by an expand path mapping directive and/or modified by transformations. For JSON source topics or TIME_SERIES topics with a JSON event type, the value option can be applied to extract part of the resulting value (the latest value in the case of TIME_SERIES topics).

A topic value option begins with the keyword as and is followed by a value directive.

A value directive is delimited by angle brackets (<, >), and consists of the value keywords and a single JSON pointer parameter. The JSON pointer selects the part of the source value to copy. For example, given a source value of:
{
"account" : "1234",
"balance" : { "amount" : 12.57, "currency" : "USD" }
}
and the value mapping clause as <value(/balance)>, the reference topic value will be:
{
"amount" : 12.57,
"currency" : "USD"
}

Value mappings that follow expand directives and/or transformations apply to the current derived value and not the source topic value.

Topic value mappings only alter the reference topic value; only the path mapping determines whether a reference topic should exist. If the topic value mapping's JSON pointer fails to select anything from the source topic value, the reference topic will have the JSON value null.

Topic value mappings are often used with path value mappings to avoid repeating information in the path and the value. For example:
map ?accounts// to balances/<scalar(/account)> as <value(/balance)>

Throttle option

The throttle option can be used to constrain the rate at which a reference topic is updated when its source topic is updated. The primary application of a throttle option is to restrict the number of updates sent to reference topic subscribers, reducing network utilization or the processing each subscriber must do. Throttling also restricts the rate at which client sessions can observe changes to reference topic values using the fetch API.

The throttle option has the form throttle to X updates every period, where X is a positive integer, and period is a positive integer followed by a time unit which is one of seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most twice every five seconds:
map ?a// to b/<path(1)> throttle to 2 updates every 5 seconds
To improve readability, the throttle option allows 1 update as an alternative to 1 updates, and every second as an alternative to every 1 seconds (and so on, for other time units). For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but updates the value of each reference topic at most once every hour:
map ?a// to b/<path(1)> throttle to 1 update every minute

The throttle option is only applied when a source topic is updated more frequently than the configured rate. If a source topic is updated less frequently, updates are passed on unconstrained. If the rate is exceeded, a reference topic will not be updated again until the configured period has expired. At this time, the reference topic will be updated based on the source topic updates that happened in the interim, and a single value will be published. Thus, the throttle option provides topic-scoped conflation.

The throttle option is ignored for time series topics because time series updates do not support efficient conflation. Updates to source time series topics are passed on immediately to the corresponding reference topics, regardless of any throttle clause.

Delay option

The delay option causes a change to a view's source topic to be delayed by a fixed time before it is reflected in reference topics. Topic additions, updates, and removals are all delayed. Delays can range from one second to many days​.

Such a publication delay is a useful way to devalue topic data so it can be given away to non-paying users​.

The delay option has the form delay by duration, where duration is a positive integer followed by a time unit which is one of seconds, minutes, or hours.

For example, the following topic view specification maps all topics below the path a to reference topics below the path b, but changes to a source topic are delayed by five minutes before they are reflected in the corresponding reference topic.
map ?a// to b/<path(1)> delay by 5 minutes

Views with the delay option specified initially create reference topics in an unpublished state. The topics are published once the delay time has expired.

Note that if you restart the Diffusion server, the delay time will need to pass before any reference topics are published. Delayed events are not persisted.

A topic in the unpublished state prevents a lower priority topic view from creating a reference topic with the same path. Sessions with the rights to read the source topic can browse unpublished topics using the withUnpublishedDelayedTopics fetch request option.

Separator option

Views can use the scalar and expand directives in path mappings to extract text from the source value. By default, any / characters in the text are interpreted as path separators and will introduce extra levels in reference topic paths. If this is undesirable, the separator option can be used to replace any / characters.

For example, consider the topic view:
map ?a/path/ to b/<scalar(/markets/name)>

Suppose the value at /markets/exchange in the source topic is "USD/Sterling". The / character is treated as a path separator, so the reference topic is created at b/USD/Sterling.

The separate option enables you to prevent this by specifying a replacement string to be used instead of a path separator in the names of derived reference topics.

In the above example, by adding a separator clause to the topic view:
map ?a/path/ to b/<scalar(/markets/name)> separator '%'
the reference topic is now created at b/USD%Sterling.

Note that the separator is a string, so it can be longer than a single character. It can contain / characters which will be interpreted as path separators. It must not contain empty path segments, that is //.

Preserve topics option

The default behavior for a topic view is that only the reference topics that can be derived from the current value of the source topic are maintained. This applies to views using directives that derive the path of the reference topic(s) from a value within the source topic (e.g. scalar or expand).

For example if a view uses a scalar directive to determine the path of thereference topic and the source topic is updated such that the selected scalar value changes then the previously created reference topic will be removed and a new one created.

Similarly an expand directive may create many reference topics from the source value but if the source is updated then only those reference topics that still have a corresponding value within the source value will be updated (or new ones added if not in the previous value), whilst others will be removed.

There are situations where an application may not want reference topics to be removed because of source value changes. In such cases the 'preserve topics' option may be used to indicate that all reference topics created by a view should be retained (and updated by later updates that apply to it) until either the source topic or the creating view are removed.

For example, given a source value of:
{
"species" : "diplodocus",
"exhibit" : { "id" : 137, "category" : "fossil" }
}
and a source value directive:
specimen/<scalar(/exhibit/category)>/species/<scalar(/species)>
the resulting reference topic path is:
specimen/fossil/species/diplodocus

What happens when the source topic value changes? By default, the reference topic at the old path is removed, and a new one is created.

In the example above, suppose the value for "species" changes from "diplodocus" to "brontosaurus".

By default, the topic at specimen/fossil/species/diplodocus is removed and a new topic is created at specimen/fossil/species/brontosaurus.

In some cases, you may not want your application's reference topics to be removed because the source value changes.

The preserve topics clause changes the default behavior so that all reference topics created by a view are retained and updated, until either the source topic or the creating view are removed.

To apply the preserve topics clause, add preserve topics to the specification, for example:
specimen/<scalar(/exhibit/category)>/species/<scalar(/species)> preserve topics
Note: Currently, preserved topics are not restored if the Diffusion server is restarted.
Note: Currently, preserved topics are not replicated across a cluster. If a new server enters a cluster, it will only have reference topics generated from the point in time where it joined the cluster and will not reflect reference topics previously created within other cluster peers.

Topic type option

The 'type' option can be used to specify the topic type of the target reference topic. If the current source value of type indicated by the source topic's type can be converted to the target type, a reference topic of the specified type will be created.

For example:
map ?a/ to b/<path(1)> type STRING

The specified type must be one of: STRING, INT64, DOUBLE, JSON, TIME_SERIES or BINARY, case insensitive.

The following table describes the supported conversions from the source topic type (the left column) to the supported target types. The number in brackets indicates a note at the foot of the table describing exactly how the conversion is processed. Where there is no number, no conversion is necessary and the derived source value is simply mapped to the target reference topic as if the 'type' option was not specified. Where there is no entry for the source topic type in the left column, no conversion is supported and topics of unsupported types will not be processed by the topic view.

In all cases, the value being processed is the current value as derived from other mappings within the topic view (for example, expand), which is not necessarily the value of the source topic.

Table 4. Reference topic type mapping
Source topic Reference topic
  STRING INT64 DOUBLE JSON TIME_SERIES BINARY
STRING - Conversion occurs if string's value can be converted to target number type. If string cannot be converted, no reference topic is created. Creates JSON topic containing the scalar value. Creates a time series with event type matching source. Source topic updates are appended to reference topic. x
INT64 Conversion from number types to STRING results in a simple string representation of the number. - INT64 to DOUBLE conversions perform a simple conversion. For example, 123 becomes 123.0. Creates JSON topic containing the scalar value. Creates a time series with event type matching source. Source topic updates are appended to reference topic. x
DOUBLE Conversion from number types to STRING results in a simple string representation of the number. DOUBLE to INT64 conversions perform rounding to the nearest integer value. For example 12.51 becomes 13. - Creates JSON topic containing the scalar value. Creates a time series with event type matching source. Source topic updates are appended to reference topic. x
JSON JSON to primitive type conversions only occur if the JSON value is a scalar which can be read as a string and converted to the target type. Currently, string and integer scalar values can be read as a string, but not doubles. The string representation is converted in the same way as specified for STRING to other primitive types. If the JSON value is a structure or cannot be converted, no reference topic is created. - Creates a time series with event type matching source. Source topic updates are appended to reference topic. x
TIME_SERIES The conversion of TIME_SERIES to other types follows the rules for the source topic type that matches the time series topic's event type. So if the time series event type is 'double', the conversion rules from DOUBLE to the target type apply. Each value appended to the source time series topic results in an update to the reference topic. If conversion fails at any point, the reference topic is removed and only recreated if a value that can be converted is appended. - BINARY to TIME_SERIES and TIME_SERIES to BINARY are supported in the same way as for other time series conversions.
BINARY x x x x BINARY to TIME_SERIES and TIME_SERIES to BINARY are supported in the same way as for other time series conversions. -
Note: An 'x' indicates that a conversion is not supported.

Escaping and quoting special characters

Each part of a topic view expression has characters with special significance. Source topic clauses and path mapping clauses are delimited by white space. Directives in path and topic property mapping clauses are delimited by the < and > characters, and each directive parameter is terminated by , or ). Topic property mapping clauses are delimited by white space, and the : and , characters.

Sometimes a topic view must refer to or generate topics with paths that containing special characters, or use a JSON pointer containing special characters. The escape sequence \x can be used to literally insert any character x, with a one exception: \/ cannot be used in path fragments since the path delimiter / is always significant.

Here is an example topic view expression containing escape sequences. It maps the topic path a topic a reference topic with the path another topic.
map a\ topic to another\ topic
Here is an example with a source value directive that uses the JSON pointer /x()/y to extract the target path from the source value. The ) character in the JSON pointer must be escaped so it is not treated as the end of the parameter list.
map ?a// to <scalar(/x(\)/y)>

To insert \, the escape sequence \\ must be used.

There is no need to escape white space in JSON pointers directive parameters. However, white space is significant. For example, the following expressions have different topic value mapping clauses since the JSON pointer in the second expression is /x ; that is, it has a trailing space:
map a to b as <value(/x)>
map a to b as <value(/x )>
Instead of using escape sequences, white space characters can be included in source topic clauses and path mapping clauses using quotes. A clause is quoted by wrapping it in single quote (') or double quote (") characters. For example:
map "a topic" to "another topic"
Within a quoted clause, quotes of the same type must be escaped:
map 'alice\'s topic' to 'bob\'s topic'

For consistency, the values in topic property mapping clauses can be escaped or quoted. However, there is no need to do so because none of the valid values for the mappable properties contain special characters.

Dealing with topic path conflicts

If you create a topic view which tries to make a reference topic with the same path as an existing topic, the result is a topic path conflict.

Reference topics have a lower priority than normal topics created through the API, including replicas of normal topics created by topic replication or fan-out. A reference topic will only be created if no topic or reference topic is already bound to its derived topic path.

Topic views have a precedence based on order of creation. If two topic views define mappings the same topic path, the earliest-created topic view will create a reference topic. If a topic view is updated, it retains its original precedence.

Remote topic views

A remote topic view maps its source topics from a remote server.

This is indicated with a from clause:
map ?a/ from server1 to b/<path(1)>

The server name server1 in this example) refers to the name of a remote server created using the Remote Servers feature.

In this case, upon establishing a successful connection with the remote server indicated the topic view will create reference topics locally based upon the topics selected by the topic view's selector at the remote server. It is important to note that the selector only refers to topics that match it at the remote server and not on the local server and there is no reason why there could not be a source topic at the remote server that has the same path as an entirely different topic on the local server.

More than one topic view can specify the same remote server.

A remote server only makes a physical connection when it is in use, therefore the first topic view that specifies a remote server will cause it to establish a connection. Similarly, if the last topic view that uses a remote server is removed then the connection will be closed.

It is not necessary for the named remote server definition to exist before creating the topic view, as if it does not then the topic view will simply remain dormant until the remote server is created and a successful connection to the server specified in its URL is established. Similarly, if a remote server that is in use by remote topic views is removed then all of the reference topics created by the topic views will be removed and the topic views will become dormant until the named remote server is created again or the views are changed to name a different remote server.

If a remote topic view selects a ROUTING topic at the remote server then local mappings will only be performed if the routing topic mapping at the remote server is able to establish a mapping for the remote server connection. The mapping will be done as if from the resolved routing topic.

The rules of precedence for remote topic views are the same as for other topic views. If the remote server for a remote topic view does not exist or does not have an established connection then the remote topic view is not evaluated (i.e. it is as if the source topics for the view did not exist), but if the remote server later connects then the view will be evaluated and rules of precedence will determine whether reference topic will replace those created by earlier views.

In a Diffusion cluster, both topic views and remote server definitions are automatically distributed across the cluster. Each member of the cluster will automatically connect to the same remote server and produce the same reference topics from the remote topic views.

All of the other topic view capabilities can be applied to a remote topic view: for example, a remote topic view can have an expand value directive or a throttle clause.

Topic view persistence and replication

Reference topics are neither replicated nor persisted. They are created and removed based on their source topics. However, topic views are replicated and persisted. A server that restarts will restore topic views during recovery. Each topic view will then create reference topics based on the source topics that have been recovered.

The server records all changes to topic views in a persistent store. Topic views are restored if the server is started.

If a server belongs to a cluster, topic views will be replicated to each server in the cluster. Topic views are evaluated locally within a server. Replicated topic views that select non-replicated source topics can create different reference topics on each server in the cluster.

Access control

The following access control restrictions are applied:
  • To list the topic views, a session needs the READ_TOPIC_VIEWS global permission.
  • To create, replace, or remove a topic view, a session needs the MODIFY_TOPIC_VIEWS global permission and SELECT_TOPIC permission for the path prefix of the source topic selector.
  • Each topic view records the principal and security roles of the session that created it as the topic view security context. When a topic view is evaluated, this security context is used to constrain the creation of reference topics. A reference topic will only be created if the security context has READ_TOPIC permission for the source topic path, and MODIFY_TOPIC permission for the reference topic path. The topic view security context is copied from the creating session at the time the topic view is created or replaced, and is persisted with the topic view. The topic view security context is not updated if the roles associated with the session are changed.