Workflow data handling

Serverless Workflow data is represented in JSON format. Data flow and execution logic go hand in hand, meaning as workflow execution follows the workflow definition logic, so does the workflow data:

Serverless Workflow Data Flow

The initial Workflow data input is passed to the workflow starting state as its data input. When a state finishes its execution, its data output is passed as data input to the next state that should be executed.

When workflow execution ends, the last executed workflow state's data output becomes the final Workflow data output.

States can filter their data inputs and outputs using State Data filters.

States can also consume events as well as invoke services. These event payloads and service invocation results can be filtered using Event data filters and Action data filters.

Data filters use workflow expressions for selecting and manipulating state data input and output, action inputs and results, and event payloads.

Multiple filters can be combined to gain high level of control of your workflow state data. You can find an example of that in this section.

Data from consumed events,and action execution results are added/merged to state data. Reference the data merging section to learn about the merging rules that should be applied.

Workflow Data Input

The initial data input into a workflow instance. Must be a valid JSON object. If no input is provided, the default data input should be an empty JSON object:

{ }

Workflow data input is passed to the workflow starting state as its data input.

Workflow data input

Information Passing Between States

States in a workflow can receive data (data input) and produce a data result (data output). The state's data input is typically the previous state's data output. When a state completes its execution, its data output is passed to the state's data input it transitions to. There are two rules to consider here:

• If the state is the workflow starting state, its data input is the workflow data input.

• When workflow execution ends, the data output of the last executed state becomes the workflow data output.

Basic state data passing

Workflow data output

Each workflow execution should produce a data output.

The workflow data output is the data output of the last executed workflow state.

State data filters

Parameter	Description	Type	Required
input	Workflow expression to filter the states data input	string	no
output	Workflow expression that filters the states data output	string	no

Example:

JSON	YAML
{ "stateDataFilter": { "input": "${ .orders }", "output": "${ .provisionedOrders }" } }	stateDataFilter: input: "${ .orders }" output: "${ .provisionedOrders }"

State data filters can be used to filter the state's data input and output.

The state data filters input property expression is applied when the workflow transitions to the current state and receives its data input. It can be used to select only data that is needed and disregard what is not needed. If input is not defined or does not select any parts of the state's data input, its data input is not filtered.

The state data filter output property expression is applied right before the state transitions to the next state defined. It filters the state's data output to be passed as data input to the transitioning state. If the current state is the workflow end state, the filtered state's data output becomes the workflow data output. If output is not defined or does not select any parts of the state's data output, its data output is not filtered.

Results of the input expression should become the state data input. Results of the output expression should become the state data output.

For more information on this you can reference the data merging section.

Let's take a look at some examples of state filters. For our examples let's say the data input to our state is as follows:

{
  "fruits": [ "apple", "orange", "pear" ],
  "vegetables": [
    {
      "veggieName": "potato",
      "veggieLike": true
    },
    {
      "veggieName": "broccoli",
      "veggieLike": false
    }
  ]
}

For the first example, our state only cares about fruits data, and we want to disregard the vegetables. To do this we can define a state filter:

{
  "stateDataFilter": {
    "input": "${ {fruits: .fruits} }"
  }
}

The state data output then would include only the fruits data:

{
  "fruits": [ "apple", "orange", "pear"]
}

State Data Filter Example

For our second example, let's say that we are interested in the only vegetable "veggie-like". Here we have two ways of filtering our data, depending on if actions within our state need access to all vegetables, or only the ones that are "veggie-like".

The first way would be to use both "input", and "output":

{
  "stateDataFilter": {
    "input": "${ {vegetables: .vegetables} }",
    "output": "${ {vegetables: [.vegetables[] | select(.veggieLike == true)]} }"
  }
}

The states data input filter selects all the vegetables from the main data input. Once all actions have performed, before the state transition or workflow execution completion (if this is an end state), the "output" of the state filter selects only the vegetables which are "veggie like".

State Data Filter Example

The second way would be to directly filter only the "veggie like" vegetables with just the data input path:

{
  "stateDataFilter": {
    "input": "${ {vegetables: [.vegetables[] | select(.veggieLike == true)]} }"
  }
}

Action data filters

Parameter	Description	Type	Required
fromStateData	Workflow expression that filters state data that can be used by the action	string	no
useResults	If set to false, action data results are not added/merged to state data. In this case 'results' and 'toStateData' should be ignored. Default is true.	boolean	no
results	Workflow expression that filters the actions data results	string	no
toStateData	Workflow expression that selects a state data element to which the action results should be added/merged into. If not specified denotes the top-level state data element	string	no

Example:

JSON	YAML
{ "actionDataFilter": { "fromStateData": "${ .language }", "results": "${ .results.greeting }", "toStateData": "${ .finalgreeting }" } }	actionDataFilter: fromStateData: "${ .language }" results: "${ .results.greeting }" toStateData: "${ .finalgreeting }"

Action data filters can be used inside Action definitions. Each action can define this filter which can:

• Filter the state data to select only the data that can be used within function definition arguments using its fromStateData property.

• Filter the action results to select only the result data that should be added/merged back into the state data using its results property.

• Select the part of state data which the action data results should be added/merged to using the toStateData property.

To give an example, let's say we have an action which returns a list of breads and pasta types. For our workflow, we are only interested into breads and not the pasta.

Action results:

{
  "breads": ["baguette", "brioche", "rye"],
  "pasta": [ "penne",  "spaghetti", "ravioli"]
}

We can use an action data filter to filter only the breads data:

{
"actions":[
    {
       "functionRef": "breadAndPastaTypesFunction",
       "actionDataFilter": {
          "results": "${ {breads: .breads} }"
       }
    }
 ]
}

The results will filter the action results, which would then be:

{
  "breads": [
    "baguette",
    "brioche",
    "rye"
  ]
}

Now let's take a look at a similar example (same expected action results) and assume our current state data is:

{
  "itemsToBuyAtStore": [
  ]
}

and have the following action definition:

{
"actions":[
    {
       "name": "fetch-items-to-buy",
       "functionRef": "breadAndPastaTypesFunction",
       "actionDataFilter": {
          "results": "${ [ .breads[0], .pasta[1] ] }",
          "toStateData": "${ .itemsToBuyAtStore }"
       }
    }
 ]
}

In this case, our results select the first bread and the second element of the pasta array. The toStateData expression then selects the itemsToBuyAtStore array of the state data to add/merge these results into. With this, after our action executes the state data would be:

{
  "itemsToBuyAtStore": [
    "baguette",
    "spaghetti"
  ]
}

In the case action results should not be added/merged to state data, we can set the useResults property to false. In this case, the results and toStateData properties should be ignored, and nothing is added/merged to state data. If useResults is not specified (or it's value set to true), action results, if available, should be added/merged to state data.

Event data filters

Parameter	Description	Type	Required
useData	If set to false, event payload is not added/merged to state data. In this case 'data' and 'toStateData' should be ignored. Default is true.	boolean	no
data	Workflow expression that filters the event data (payload)	string	no
toStateData	Workflow expression that selects a state data element to which the action results should be added/merged into. If not specified denotes the top-level state data element	string	no

Example:

JSON	YAML
{ "eventDataFilter": { "data": "${ .data.results }" } }	eventDataFilter: data: "${ .data.results }"

Event data filters can be used to filter consumed event payloads. They can be used to:

• Filter the event payload to select only the data that should be added/merged into the state data using its data property.

• Select the part of state data into which the event payload should be added/merged into using the toStateData property.

Allows event data to be filtered and added to or merged with the state data. All events have to be in the CloudEvents format and event data filters can filter both context attributes and the event payload (data) using the data property.

Here is an example using an event filter:

Event Data Filter Example

Note that the data input to the Event data filters depends on the dataOnly property of the associated Event definition. If this property is not defined (has default value of true), Event data filter expressions are evaluated against the event payload (the CloudEvents data attribute only). If it is set to false, the expressions should be evaluated against the entire CloudEvent (including its context attributes).

In the case event data/payload should not be added/merged to state data, we can set the useData property to false. In this case, the data and toStateData properties should be ignored, and nothing is added/merged to state data. If useData is not specified (or it's value set to true), event payload, if available, should be added/merged to state data.

Using multiple data filters

As Event states can take advantage of all defined data filters. In the example below, we define a workflow with a single event state and show how data filters can be combined.

{
    "name": "greet-customers-workflow",
    "description": "Greet Customers when they arrive",
    "version": "1.0.0",
    "specVersion": "0.8",
    "start": "WaitForCustomerToArrive",
    "states":[
         {
            "name": "wait-for-customer-to-arrive",
            "type": "event",
            "onEvents": [{
                "eventRefs": ["customer-arrives-event"],
                "eventDataFilter": {
                    "data": "${ .customer }",
                    "toStateData": "${ .customerInfo }"
                },
                "actions":[
                    {
                        "name": "greet-customer",
                        "functionRef": {
                            "refName": "greeting-function",
                            "arguments": {
                                "greeting": "${ .hello.spanish } ",
                                "customerName": "${ .customerInfo.name } "
                            }
                        },
                        "actionDataFilter": {
                            "fromStateData": "${ { hello, customerInfo } }",
                            "results": "${ .greetingMessageResult }",
                            "toStateData": "${ .finalCustomerGreeting }"
                        }
                    }
                ]
            }],
            "stateDataFilter": {
                "input": "${ .greetings } ",
                "output": "${ { finalCustomerGreeting } }"
            },
            "end": true
        }
    ],
    "events": [{
        "name": "customer-arrives-event",
        "type": "customer-arrival-type",
        "source": "customer-arrival-event-source"
     }],
    "functions": [{
        "name": "greeting-function",
        "operation": "http://my.api.org/myapi.json#greeting"
    }]
}

The workflow data input when starting workflow execution is assumed to include greetings in different languages:

{
  "greetings": {
      "hello": {
        "english": "Hello",
        "spanish": "Hola",
        "german": "Hallo",
        "russian": "Здравствуйте"
      },
      "goodbye": {
        "english": "Goodbye",
        "spanish": "Adiós",
        "german": "Auf Wiedersehen",
        "russian": "Прощай"
      }
  }
}

The workflow data input then becomes the data input of the starting workflow state.

We also assume for this example that the CloudEvent that our event state consumes include the data (payload):

{
 "customer": {
   "name": "John Michaels",
   "address": "111 Some Street, SomeCity, SomeCountry",
   "age": 40
 }
}

Here is a sample diagram showing our workflow, each numbered step on this diagram shows a certain defined point during workflow execution at which data filters are invoked and correspond to the numbered items below.

Using Multple Filters Example

(1) Workflow execution starts: Workflow data is passed to our "WaitForCustomerToArrive" event state as data input. Workflow executes its starting state, namely the "WaitForCustomerToArrive" event state.

The event state stateDataFilter is invoked to filter its data input. The filters "input" expression is evaluated and selects only the "greetings" data. The rest of the state data input should be disregarded.

At this point our state data should be:

{
  "hello": {
    "english": "Hello",
    "spanish": "Hola",
    "german": "Hallo",
    "russian": "Здравствуйте"
  },
  "goodbye": {
    "english": "Goodbye",
    "spanish": "Adiós",
    "german": "Auf Wiedersehen",
    "russian": "Прощай"
  }
}

(2) CloudEvent of type "customer-arrival-type" is consumed: Once the event is consumed, the "eventDataFilter" is triggered. Its "data" expression selects the "customer" object from the events data. The "toStateData" expression says that we should add/merge this selected event data to the state data in its "customerInfo" property. If this property exists it should be merged, if it does not exist, one should be created.

At this point our state data contains:

{
    "hello": {
      "english": "Hello",
      "spanish": "Hola",
      "german": "Hallo",
      "russian": "Здравствуйте"
    },
    "goodbye": {
      "english": "Goodbye",
      "spanish": "Adiós",
      "german": "Auf Wiedersehen",
      "russian": "Прощай"
    },
    "customerInfo": {
       "name": "John Michaels",
       "address": "111 Some Street, SomeCity, SomeCountry",
       "age": 40
     }
}

(3) Event state performs its actions: Before the first action is executed, its actionDataFilter is invoked. Its "fromStateData" expression filters the current state data to select from its data that should be available to action arguments. In this example it selects the "hello" and "customerInfo" properties from the current state data. At this point the action is executed. We assume that for this example "greetingFunction" returns:

{
   "execInfo": {
     "execTime": "10ms",
     "failures": false
   },
   "greetingMessageResult": "Hola John Michaels!"
}

After the action is executed, the actionDataFilter "results" expression is evaluated to filter the results returned from the action execution. In this case, we select only the "greetingMessageResult" element from the results.

The action filters "toStateData" expression then defines that we want to add/merge this action result to state data under the "finalCustomerGreeting" element.

At this point, our state data contains:

{
  "hello": {
      "english": "Hello",
      "spanish": "Hola",
      "german": "Hallo",
      "russian": "Здравствуйте"
    },
    "goodbye": {
      "english": "Goodbye",
      "spanish": "Adiós",
      "german": "Auf Wiedersehen",
      "russian": "Прощай"
    },
    "customerInfo": {
       "name": "John Michaels",
       "address": "111 Some Street, SomeCity, SomeCountry",
       "age": 40
     },
     "finalCustomerGreeting": "Hola John Michaels!"
}

(4) Event State Completes Execution:

When our event state finishes its execution, the states "stateDataFilter" "output" filter expression is executed to filter the state data to create the final state data output.

Because our event state is also an end state, its data output becomes the final workflow data output. Namely:

{
   "finalCustomerGreeting": "Hola John Michaels!"
}

Data Merging

Consumed event data (payload) and action execution results should be merged into the state data. Event and action data filters can be used to give more details about this operation.

By default, with no data filters specified, when an event is consumed, its entire data section (payload) should be merged to the state data. Merging should be applied to the entire state data JSON element.

In case of event and action filters, their "toStateData" property can be defined to select a specific element of the state data with which merging should be done against. If this element does not exist, a new one should be created first.

When merging, the state data element and the data (payload)/action result should have the same type, meaning that you should not merge arrays with objects or objects with arrays etc.

When merging elements of type object should be done by inserting all the key-value pairs from both objects into a single combined object. If both objects contain a value for the same key, the object of the event data/action results should "win". To give an example, let's say we have the following state data:

{
    "customer": {
        "name": "John",
        "address": "1234 street",
        "zip": "12345"
    }
}

and we have the following event payload that needs to be merged into the state data:

{
    "customer": {
        "name": "John",
        "zip": "54321"
    }
}

After merging the state data should be:

{
  "customer": {
    "name": "John",
    "address": "1234 street",
    "zip": "54321"
  }
}

Merging array types should be done by concatenating them into a larger array including unique elements of both arrays. To give an example, merging:

{
    "customers": [
      {
        "name": "John",
        "address": "1234 street",
        "zip": "12345"
      },
      {
        "name": "Jane",
        "address": "4321 street",
        "zip": "54321"
      }
    ]
}

into state data:

{
    "customers": [
      {
        "name": "Michael",
        "address": "6789 street",
        "zip": "6789"
      }
    ]
}

should produce state data:

{
    "customers": [
      {
        "name": "Michael",
        "address": "6789 street",
        "zip": "6789"
      },
      {
        "name": "John",
        "address": "1234 street",
        "zip": "12345"
      },
      {
        "name": "Jane",
        "address": "4321 street",
        "zip": "54321"
      }
    ]
}

Merging number types should be done by overwriting the data from events data/action results into the merging element of the state data. For example merging action results:

{
    "age": 30
}

into state data:

{
    "age": 20
}

would produce state data:

{
    "age": 30
}

Merging string types should be done by overwriting the data from events data/action results into the merging element of the state data.