time-series.h File Reference

This feature allows a session to update and query time series topics. More...

Data Structures
struct	diffusion_time_series_append_params_s
	Structure supplied when issuing a diffusion_time_series_append request. More...
struct	diffusion_time_series_timestamp_append_params_s
	Structure supplied when issuing a diffusion_time_series_timestamp_append request. More...
struct	diffusion_time_series_edit_params_s
	Structure supplied when issuing a diffusion_time_series_edit request. More...

Typedefs
typedef struct DIFFUSION_TIME_SERIES_EVENT_METADATA_T	DIFFUSION_TIME_SERIES_EVENT_METADATA_T
	Opaque diffusion time series event metadata struct.
typedef struct DIFFUSION_TIME_SERIES_EVENT_T	DIFFUSION_TIME_SERIES_EVENT_T
	Opaque diffusion time series event struct.
typedef int(*	on_append_cb )(const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata, void *context)
	Callback when a time series append operation is successful. More...
typedef int(*	on_edit_cb )(const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata, void *context)
	Callback when a time series edit operation is successful. More...
typedef struct diffusion_time_series_append_params_s	DIFFUSION_TIME_SERIES_APPEND_PARAMS_T
	Structure supplied when issuing a diffusion_time_series_append request.
typedef struct diffusion_time_series_timestamp_append_params_s	DIFFUSION_TIME_SERIES_TIMESTAMP_APPEND_PARAMS_T
	Structure supplied when issuing a diffusion_time_series_timestamp_append request.
typedef struct diffusion_time_series_edit_params_s	DIFFUSION_TIME_SERIES_EDIT_PARAMS_T
	Structure supplied when issuing a diffusion_time_series_edit request.

Functions
long	diffusion_time_series_event_metadata_get_sequence (const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata)
	Return the sequence number identifying this event within its time series. More...
long	diffusion_time_series_event_metadata_get_timestamp (const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata)
	Return the event timestamp. More...
char *	diffusion_time_series_event_metadata_get_author (const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata)
	Returns a memory allocated string of the server-authenticated identity of the session that created the event. More...
DIFFUSION_TIME_SERIES_EVENT_METADATA_T *	diffusion_time_series_event_metadata_dup (const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata)
	Returns a memory allocated copy of a DIFFUSION_TIME_SERIES_EVENT_METADATA_T More...
void	diffusion_time_series_event_metadata_free (DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata)
	Free a memory allocated DIFFUSION_TIME_SERIES_EVENT_METADATA_T More...
long	diffusion_time_series_event_get_sequence (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	Return the sequence number identifying this event within its time series. More...
long	diffusion_time_series_event_get_timestamp (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	Return the event timestamp. More...
char *	diffusion_time_series_event_get_author (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	Returns a memory allocated string of the server-authenticated identity of the session that created the event. More...
DIFFUSION_VALUE_T *	diffusion_time_series_event_get_value (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	Returns a memory allocated DIFFUSION_VALUE_T, the time series event's value. More...
DIFFUSION_TIME_SERIES_EVENT_METADATA_T *	diffusion_time_series_event_get_original_event (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	If this is an edit event, returns the metadata of the original event that this event replaces; otherwise returns this event. More...
int	diffusion_time_series_event_is_edit_event (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	Return whether this is an edit event. More...
DIFFUSION_TIME_SERIES_EVENT_T *	diffusion_time_series_event_with_value (const DIFFUSION_TIME_SERIES_EVENT_T *event, const BUF_T *value)
	Clone this event with a different value. More...
bool	read_diffusion_time_series_event (const DIFFUSION_VALUE_T *value, DIFFUSION_TIME_SERIES_EVENT_T **event, DIFFUSION_API_ERROR *error)
	Get the DIFFUSION_TIME_SERIES_EVENT_T update value. More...
DIFFUSION_TIME_SERIES_EVENT_T *	diffusion_time_series_event_dup (const DIFFUSION_TIME_SERIES_EVENT_T *event)
	Returns a memory allocated copy of a DIFFUSION_TIME_SERIES_EVENT_T. More...
void	diffusion_time_series_event_free (DIFFUSION_TIME_SERIES_EVENT_T *event)
	Free a memory allocated DIFFUSION_TIME_SERIES_EVENT_T More...
bool	diffusion_time_series_append (SESSION_T *session, const DIFFUSION_TIME_SERIES_APPEND_PARAMS_T params, DIFFUSION_API_ERROR *api_error)
	Update a time series topic by appending a new value. More...
bool	diffusion_time_series_timestamp_append (SESSION_T *session, const DIFFUSION_TIME_SERIES_TIMESTAMP_APPEND_PARAMS_T params, DIFFUSION_API_ERROR *api_error)
	Update a time series topic by appending a new value with a supplied timestamp. More...
bool	diffusion_time_series_edit (SESSION_T *session, const DIFFUSION_TIME_SERIES_EDIT_PARAMS_T params, DIFFUSION_API_ERROR *api_error)
	Update a time series topic by appending a new value that overrides the value of an existing event. More...
DIFFUSION_TIME_SERIES_RANGE_QUERY_T *	diffusion_time_series_range_query (void)
	Return a default range query that performs a value range query of an entire time series. More...
void	diffusion_time_series_range_query_free (DIFFUSION_TIME_SERIES_RANGE_QUERY_T *range_query)
	Free a memory allocated DIFFUSION_TIME_SERIES_RANGE_QUERY_T More...

Detailed Description

This feature allows a session to update and query time series topics.

Time series topics

A time series is a sequence of events. Each event contains a value and has server-assigned metadata comprised of a sequence number, timestamp, and author. Events in a time series are ordered by increasing sequence number. Sequence numbers have values between 0 and LONG_MAX and are contiguous: an event with sequence number n will be followed by one with sequence number n + 1. Two events with the same sequence number will be equal; having the same timestamp, author, and value.

A time series topic allows sessions to access a time series that is maintained by the server. A time series topic has an associated event data type, such as Binary, String, or JSON, that determines the type of value associated with each event.

This feature provides a historic query API for time series topics, allowing a session to query arbitrary sub-sequences of a time series. The topic-control and topics features complete the API, providing ways to create and subscribe to a time series topic.

The API presents a time series as an append-only data structure of immutable events that is only changed by adding new events.

Edit events

Although a time series is append-only, an event can be overridden by appending an edit event. An edit event is a special type of event that overrides an earlier event in the time series (referred to as the original event) with a new value. When an edit event is added to a time series, the server retains both the original event and the edit event, allowing subscription and query results to reflect the edit.

For example, suppose a time series has two events with the values A and B, and the first event has been overridden by a later edit event that provides a new value of X. The server has the following information about the time series.

Sequence	Value	Type
0	A	original event
1	B	original event
2	X	edit of sequence 0

The current value of the event with sequence number 0 is X.

If an original event has several edit events, the latest edit event (the one with the highest sequence number) determines its current value. Each edit event refers to an original event, never to another edit event.

Extending the example by appending a further edit event to the time series:

Sequence	Value	Type
3	Y	second edit of sequence 0

The current value of the event with sequence number 0 is now Y.

Retained range

A time series topic retains a range of the most recent events. When a new event is added to the time series, older events that fall outside of the range are discarded. By default, this range includes the ten most recent events. A different range can be configured by setting the DIFFUSION_TIME_SERIES_RETAINED_RANGE property.

Subscribing to a time series topic

A session can subscribe to a time series topic and add a value stream to receive updates about events appended to the time series. Events are represented by DIFFUSION_TIME_SERIES_EVENT_T instances. Each event has a value and DIFFUSION_TIME_SERIES_EVENT_METADATA_T. An edit event has two sets of metadata; its own metadata and that of the original event that it replaces.

Subscription range

New subscribers are sent a range of events from the end of the time series. This is known as the subscription range. Configuring a subscription range is a convenient way to provide new subscribers with an appropriate subset of the latest events.

The default subscription range depends on whether the topic is configured to publish delta streams. If delta streams are enabled, new subscribers are sent the latest event if one exists. If delta streams are disabled, new subscribers are sent no events. Delta streams are enabled by default and can be disabled by setting the DIFFUSION_PUBLISH_VALUES_ONLY property to "true".

A larger subscription range can be configured by setting the DIFFUSION_TIME_SERIES_SUBSCRIPTION_RANGE property. Regardless of the DIFFUSION_TIME_SERIES_SUBSCRIPTION_RANGE property, if delta streams are enabled, new subscribers will be sent at least the latest event if one exists.

If the range of events is insufficient, the subscribing session can use a range query to retrieve older events.

When configuring a non-default subscription range for a time series topic, register value streams before subscribing to the topic. The session only maintains a local cache of the latest value received for a topic, not the full subscription range. If a value stream is added after a session has subscribed to a matching time series topic, the new stream will only be notified of the latest value.

Updating a time series topic

A session can use diffusion_time_series_append to submit a value to be added to a time series. The server will add an event to the end of the time series based on the supplied value, with a new sequence number, timestamp, and the author set to the authenticated principal of the session.

A session can use diffusion_time_series_timestamp_append to submit a value and supplied timestamp. This provides control over the timestamp of the event. The supplied timestamp must not be before the latest event stored by the time series topic. There are no other restrictions.

A session can use diffusion_time_series_edit to submit an edit to an original time series event, identified by its sequence number. The server will add an edit event to the end of the time series based on the supplied value, with a new sequence number, timestamp, and the author set to the authenticated principal of the session.

Time series topics can also be updated using the functionality provided by the Topic Update feature. This includesdiffusion_topic_update_set, diffusion_topic_update_add_and_set and update streams. This usage performs an append operation with the added benefits of update constraints, topic creation when updating (upsert), and delta streams. When using methods from Topic Update, the sequence number, timestamp, and author metadata will be generated using the same rules as diffusion_time_series_append but the associated DIFFUSION_TIME_SERIES_EVENT_METADATA_T will not be returned to the caller.

Querying a time series topic

A range quert is a configured query that can be evaluated for a time series topic using diffusion_time_series_select_from. Results are provided as streams of DIFFUSION_TIME_SERIES_EVENT_T instances.

DIFFUSION_TIME_SERIES_RANGE_QUERY_T is a builder for configuring a Query that selects a range of a time series. There are two types of range query that differ in how edits are processed; value range queries and edit range queries.

Value range queries

A value range query returns a merged view of part of a time series. This is the most common time series query and appropriate for most applications.

The result of a value range query reflects the latest available edits and the query result list is ordered by the original event sequence number, presenting edit events instead of the original events they replace. Original events that have no edit events are included verbatim. Original events that have edit events are replaced by the latest edit event.

A value range query of the example time series, with no range constraints so the entire time series is selected, returns two events:

sequence=3, value=Y; original event sequence=0
sequence=1, value=B

The original value of the first event is not provided. It's apparent that the first event is an edit event because it provides the metadata of the original event it replaces.

Edit range queries

Applications with auditing and other administrative requirements can access original event values using an edit range query. An edit range query returns an unmerged view of a time series that can include both original events and the edit events that replace them. Edit range queries are rarely needed – value range queries satisfy most use cases.

Edit range queries provide a detailed view of a time series. Because this is potentially sensitive information, an edit range query can only be performed by a session that has the QUERY_OBSOLETE_TIME_SERIES_EVENTS permission for the target topic.

There are two sub-types of edit range query.

A full audit trail of edit events can be obtained using an all edits edit range query. The result contains all original events selected by the query, together with all subsequent edit events that affect the original events. The query result stream provides events in time series order. An all edits query of the example time series, with no range constraints so the entire time series is selected, returns four events:

sequence=0; value=A
sequence=1; value=B
sequence=2; value=X; original event sequence=0
sequence=3; value=Y; original event sequence=0

A latest edits edit range query returns a query result stream in time series order that contains all original events selected by the query, together with the latest edit events that affect the original events. A latest edits query of the example time series, with no range constraints so the entire time series is selected, returns three events:

sequence=0; value=A
sequence=1; value=B
sequence=3; value=Y; original event sequence=0

The initial range of events delivered for a subscription to a time series topic is derived from a latest edits edit range query. See Subscription Range.

When evaluated for a time series that has no edit events, an edit range query will return the same results as a similarly configured value range query.

Changes to a time series made outside the API

The API presents a time series as an append-only data structure of immutable events that is only changed by adding new events. The API does not allow events to be deleted or edited.

There are circumstances in which events can be removed from a time series by server operations outside the API. For example, a time series topic can be configured to discard or archive older events to save storage space; or the time series may be held in memory and lost if the server restarts. Subscribed sessions are not notified when events are removed in this way, but a session can infer the removal of events that are no longer included in query results. Similarly, an event's value can be changed on the server. For example, if an administrator changes its value to redact sensitive data. Again, subscribed sessions are not notified when events are modified, but a session can infer this has happened from query results.

Whether such changes can happen for a particular time series topic depends on the topic specification, and the administrative actions that are allowed. To write a robust application, do not rely on two Event instances with the same sequence number but obtained though different API calls, being equal; nor that there are no sequence number gaps between events in query results.

Access control

The session must have the READ_TOPIC path permission for a topic to query a time series topic. The QUERY_OBSOLETE_TIME_SERIES_EVENTS path permission is additionally required to evaluate an diffusion_time_series_range_query_for_edits query, or a diffusion_time_series_range_query_for_values with a diffusion_time_series_range_query_edit_range.

The session must have the UPDATE_TOPIC path permission for a topic to append a new event to a time series topic. The EDIT_TIME_SERIES_EVENTS path permission is additionally required to submit an edit to any time series topic event. The more restrictive EDIT_OWN_TIME_SERIES_EVENTS path permission allows a session to submit edits to time series topic events that are authored by the principal of the calling session.

Typedef Documentation

typedef int(* on_append_cb)(const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata, void *context)

Callback when a time series append operation is successful.

Parameters

event_metadata	The event metadata.
context	User supplied context.

Returns

HANDLER_SUCCESS or HANDLER_FAILURE.

typedef int(* on_edit_cb)(const DIFFUSION_TIME_SERIES_EVENT_METADATA_T *event_metadata, void *context)

Callback when a time series edit operation is successful.

Parameters

event_metadata	The event metadata.
context	User supplied context.

Returns

HANDLER_SUCCESS or HANDLER_FAILURE.