Iteration and loops

Overview

In the Variant platform, there are multiple ways to perform loops or iterate over data, but they primarily fall into two categories: ForEach-style loops and While loops. This section focuses on the ForEach family of pipes built into the platform.

ForEach Loops

The Variant development language provides three core pipes for iteration:

• Variant.Core.ForEachPipe

• Variant.Core.ForEachAsyncPipe

• Variant.Core.ForEachAsyncEnumerablePipe

Each variation has different capabilities and is suited for specific use cases, primarily differing in how and where they retrieve the data to iterate over.

Common Settings

All ForEach pipes share several common configuration options:

• PATH_OR_ARRAY: A literal array or a substitution expression pointing to an enumerable header value. If not provided, ENUMERABLE_STRATEGY must be used.

• ITEM_NS: Namespace where the current item is stored in each iteration.

• START_COUNTER_AT: Optional start value for a loop counter.

• COUNTER_NAMESPACE: Header where the loop counter value is stored.

• SCOPED_PIPES: The list of pipes executed for each item.

Variant.Core.ForEachPipe

This simple pipe uses data from either an IEnumerableStrategy, an array or IEnumerable value found in the headers or a data array set directly on the pipe. It's core property settings are:

  - pipe: Variant.Core.ForEachPipe
    replacements:
      ENUMERABLE_STRATEGY: # A strategy derived from the IEnumerableStrategy interface 
      PATH_OR_ARRAY:
      ITEM_NS: Item
      START_COUNTER_AT: 
      COUNTER_NAMESPACE: ForEachPipe.Counter         
      SCOPED_PIPES: 
      # Additional settings removed for brevity

The following example just adds the array directly. But, this can be a substitution value or a strategy.

  - pipe: Variant.Core.ForEachPipe
    PATH_OR_ARRAY!:
      - firstName: Bob
        surname: Scratch
      - firstName: Iva
        surname: Itch      
    ITEM_NS!: Item
    START_COUNTER_AT!: 0     
    SCOPED_PIPES: 
      - pipe: Variant.Core.LogDebugMessage
        LOG_MESSAGE: "${ForEachPipe.Counter}->${Item.surname}

The above example uses an inline array but having the array in a header would produce the same results.

tip

It's good practice to always suffix your setting names with a '!'. This ensure that the value cannot be overwritten if you are nesting 'foreach' type pipes

Variant.Core.ForEachAsyncPipe

This pipe behaves similarly to ForEachPipe but with two key differences: iterations may run concurrently depending on the behavior of the scoped pipes, and any errors thrown during execution are aggregated and returned as an AggregatePipeResult instead of a standard PipeResult. Its core configuration options are:

  - pipe: Variant.Core.ForEachAsyncPipe
    replacements:
      ENUMERABLE_STRATEGY: # A strategy derived from the IEnumerableStrategy interface 
      PATH_OR_ARRAY: 
      ITEM_NS: Item
      START_COUNTER_AT: 
      COUNTER_NAMESPACE: ForEachAsyncPipe.Counter
      MAX_CONCURRENCY: 8
      MAX_ERROR_COUNT: 0
      SCOPED_PIPES: 
          # Additional settings removed for brevity

Variant.Core.ForEachAsyncEnumerablePipe

This pipe enables high-performance streaming by iterating over data without requiring the entire dataset to be loaded into memory. It relies on strategies that implement the IAsyncEnumerableStrategy interface. A typical use case is found in the Variant.Strategies.Csv extension package, which supports efficient processing of large CSV files. Its core configuration settings are:

  - pipe: Variant.Core.ForEachAsyncEnumerablePipe
    replacments:
      ASYNC_ENUMERABLE_STRATEGY: 
	  ITEM_NS: Item
      START_COUNTER_AT: 
      COUNTER_NAMESPACE: ForEachPipe.Counter
      BATCHED_ITEMS_COUNT: 1
      BATCHED_ITEMS_CREATE_NEW_LIST: False
      SCOPED_PIPES:

tip

If you set the BATCHED_ITEMS_COUNT to a value greater than 1 then you can use the ITEM_NS value as the input for PATH_OR_ARRAY setting in the other 2 'foreach' pipes.

Looping with For, While, and DoWhile Pipes

Sometimes you need to repeat a set of operations a fixed number of times or until a specific condition is met. Variant provides three dedicated control-flow pipes for these use cases:

• Variant.Core.ForPipe – Loops a defined number of times.

• Variant.Core.WhilePipe – Continues looping while a condition is true.

• Variant.Core.DoWhilePipe – Executes once and continues looping while a condition remains true.

Variant.Core.ForPipe

The Variant.Core.ForPipe is used when you know exactly how many times you want to repeat a block of logic. Its core configuration properties are:

  - pipe: Variant.Core.ForPipe
    replacements:
      START_AT: 
      END_AT: 
      INCREMENT_BY: 
      COUNTER_NAMESPACE: ForCounter
      SCOPED_PIPES:

An example of this pipe in action using a subsitutable END_AT value can be found in the following example:

- pipe: Variant.Core.ModifyMessageStrategyPipe
  NAMESPACE: Items
  VALUE: 
    - firstName: Ian
      surname: Hill
    - firstName: Mike
      surname: Douglas

- pipe: Variant.Core.ForPipe
  START_AT: 1
  INCREMENT_BY: 1
  END_AT: ${Items.Count()}
  COUNTER_NAMESPACE: ForCounter
  SCOPED_PIPES: 
    - pipe: Variant.Core.LogMessagePipe
      LOG_MESSAGE: Name ${Items.GetValueAt(${ForCounter}).firstName}

Which equates to:

for (var n = 0; n < ${EndAt}; n++){
    Console.WriteLine("Name " + Items[n].furstName");
}

Variant.Core.WhilePipe & Variant.Core.DoWhilePipe

Use WhilePipe or DoWhilePipe when you need to repeat a set of actions until a specific condition is met (or no longer met). Both share the same configuration:

 pipe: Variant.Core.WhilePipe
 replacements:
   BREAK_EXECUTION_STRATEGY: 
   MAX_NUMBER_OF_LOOPS: 
   BREAK_EXPRESSION: 
   SCOPED_PIPES:

The only difference between the two is that DoWhilePipe always executes the scoped pipes at least once, regardless of the initial condition. All future examples will use WhilePipe for consistency.

Examples

While(${SomeValue} == true)

- pipe: Variant.Core.WhilePipe
  BREAK_EXPRESSION: ${SomeValue} != true
  MAX_NUMBER_OF_LOOPS: -1
  SCOPED_PIPES: 
    - pipe: DoSomething Pipe

Equates to:

while(${SomeValue} == true){

    // Do Stuff
}

While(true)

- pipe: Variant.Core.WhilePipe
  MAX_NUMBER_OF_LOOPS: -1
  SCOPED_PIPES: 
    - pipe: DoSomething Pipe
      CAN_EXECUTE_EXPRESSION: ${SomeValue} == true

Equates to:

while(true){

    // Do Stuff
    
    if(${SomeValue} == true)
      break ;
}

Break Pipes

When iterating over an array, you may want to exit the entire loop prematurely. To do this, use a BreakPipe and set the EXIT_ANCESTOR_LOOP property to true.

By default, this setting is false, which means the pipe will only skip the current item and continue with the next.

Alternatively, you can use Variant.Core.LoopBreakPipe, which is a specialized version with EXIT_ANCESTOR_LOOP already set to true.