Skip to main content

Flow

A data flow represents a logical connection between two platforms within Flowlyze.
The connection occurs through a Source (data source) and a Destination (data destination).
Understanding how data flows work requires familiarity with three fundamental concepts: Message, Source, and Destination — plus the operational concept of Flow.

Message​

A Message is the elementary unit of information that transits within Flowlyze, regardless of the data type or source format.
It can represent, for example:

  • a row of a CSV file,
  • a record extracted from a relational database,
  • or the result of an HTTP call to an API.

Each Message is structured as a nested object and can be assimilated to a JSON document (JavaScript Object Notation).
This format allows Flowlyze to handle complex data flexibly and in a standardized way.

Source​

The Source is the data source, i.e., the point from which Flowlyze receives or extracts information.
It can take various forms:

  • API (Application Programming Interface),
  • Database (relational or non-relational),
  • Flat file (such as CSV or XML).

Sources can be managed in:

  • active mode, when Flowlyze directly queries the source to obtain data;
  • passive mode, when the source itself sends data to Flowlyze (e.g., via webhook).
Additional information

🔎 For more details, consult the "Source" section of the documentation.

Destination​

The Destination represents the destination of data processed by Flowlyze.
In this case too, it can be APIs, databases, or file systems, managed in active or passive mode depending on integration needs.
Flowlyze ensures that data is delivered correctly, adapting it to the format required by the destination platform.

Additional information

🔎 To learn more, consult the "Destination" section of the documentation.

Flow​

The Flow is the operational element that connects a Source to a Destination.
Within the flow, data transformation, interconnection, and adaptation logics are defined and implemented, ensuring compatibility between systems with different structures and protocols.

Flow Management and Monitoring​

Data ingested from a Source is stored and actively managed within Flowlyze.
This approach enables the system to:

  • monitor the delivery of data to their respective destinations;
  • automatically manage error notifications and retry attempts in case of anomalies;
  • ensure complete traceability of data along the entire integration path.

Through the management interface, it is possible to:

  • view the list of messages that have transited and are in transit,
  • correct and resend any undelivered data,
  • and consult transfer statistics for each individual flow.

Retry Policies​

Flowlyze implements two retry policies (automatic retransmission) for managing messages not delivered to the destination platform.
These logics ensure data flow resilience even in the presence of temporary errors or downtime of integrated systems.

1. Linear Retry​

Linear management involves a fixed number of delivery attempts (N) performed at regular and constant intervals.
It is the simplest and most predictable mode, recommended when the error cause is generally temporary and short-lived.

Operational example

A new attempt every 5 minutes, for a maximum of 10 attempts.

Main characteristics:

  • Constant interval between one retry and the next.
  • Greater predictability in flow completion times.
  • Ideal for stable systems or with low probability of prolonged downtime.

2. Exponential Retry​

Exponential management dynamically adjusts the interval between delivery attempts, increasing it progressively after each error.
This approach allows reducing pressure on the receiving system and giving it the time necessary to restore its health status.

Operational example

First retry after 2 minutes, then after 4, 16, and 32 minutes.

Main characteristics:

  • Increasing interval between retransmission attempts.

  • Greater tolerance to prolonged downtime.

  • Particularly useful in contexts with variable loads or external integrations not always available.

Configuration

Both retry modes can be configured at the Flow or Destination level, based on the specific needs of the integration project.

Message Order Management and Synchronization​

Flowlyze is an iPaaS (Integration Platform as a Service) platform based on a completely asynchronous architecture, designed to ensure reliability, scalability, and high performance in data processing between heterogeneous systems.

The platform ensures correct message delivery to destination systems, but does not guarantee the temporal order of sending or receiving them.
This approach allows maximizing performance and maintaining efficient management of processing queues even in high-load scenarios or temporary network errors.

Data Idempotency​

Important

Since message order is not guaranteed, it is essential that destination systems implement idempotency mechanisms, i.e., strategies that allow correctly handling duplicate or out-of-sequence messages.

Examples of idempotent approaches include:

  • Discarding obsolete data: the receiving system verifies the last modification date or a unique identifier and ignores messages that are no longer current.
  • Smart upserts: write operations combine insert and update, updating only if the received data is more recent than that already present.
  • Timestamp or version-based checks: each record includes a version field or a timestamp used to determine message validity.
Best practice

These precautions ensure that the data flow remains consistent even in the presence of parallel executions or delayed messages.

Parallelism and Non-Chronological Processing​

For performance and scalability reasons, Flowlyze can process messages in parallel or in non-chronological order.
This means that:

  • Data can be processed by multiple workers simultaneously, reducing overall processing times
  • In case of an error on a message, queue processing is not blocked: subsequent messages continue to be processed without interruptions.
  • Messages in error are handled through retry mechanisms, allowing controlled recovery without impacting the general flow.

Benefits of the Asynchronous Approach​

  • High reliability: each message is guaranteed delivery, even in case of temporary errors.
  • Horizontal scalability: the load can be distributed among multiple workers or processing nodes.
  • Resilience: the system remains operational even in the presence of problematic messages.
  • Efficiency: parallel processing reduces overall latency.

Buffer Management​

Flowlyze manages data sending to destination platforms through batch processing, a mode that enables optimizing performance and reducing the number of calls between systems.

Batch sending can occur, for example:

  • through a single transaction to a database, or
  • via a single HTTP call to an external system.

This approach is essential to reduce round trip — i.e., the overall request and response time between two systems — and increase overall integration efficiency.

Configuration​

Each Flow (data flow) can be configured to dynamically manage batch sending behavior.
Through configuration parameters, you can define:

  • Maximum number of items per batch:
    determines the number of messages to accumulate before proceeding with sending to the destination.
  • Maximum data accumulation time:
    establishes how long messages can remain pending before being sent, even if the maximum batch size has not been reached.
  • Maximum number of parallel sends:
    indicates the number of simultaneous sending processes that the destination system can handle, useful for load balancing and preventing saturation.

Operating Logic​

Flowlyze accumulates messages from the Source until:

  • the maximum number of items per batch is reached, or
  • the maximum accumulation time configured expires.

Upon expiration of either condition, the system automatically initiates the sending procedure to the destination platform.

Configuration Example​

Example scenario

Scenario: sending a maximum of 1,000 messages with a 30-second timeout.
Result: Flowlyze will send data as soon as:

  • the number of 1,000 messages is reached, or
  • 30 seconds have elapsed since the last send, regardless of the number of messages collected.

Message Status Flow​

During their operational path, messages processed by Flowlyze go through a series of states that describe their evolution, from initial reception to delivery (or their eventual discard).
This state-based management enables precise monitoring of data flow advancement and timely intervention in case of errors or operational blocks.

New​

The message has been received by the system but not yet processed.
In this state, the data is waiting to be taken over by the sending engine.

Processing​

The message is being processed.

It has been taken over by the sending engine but has not yet been transmitted to the destination platform.

Sent​

The message has been forwarded to the destination and is awaiting feedback.
Flowlyze awaits confirmation of receipt or the outcome of the request (success or error).

Success​

The message has been correctly delivered to the destination platform.
This represents a final and positive state of the flow.

Error​

The message was not delivered due to a destination-side error (e.g., an unreachable HTTP endpoint or a validation error).
With active retry policies, Flowlyze will attempt new automatic deliveries according to the configured logic (linear or exponential).

Aborted​

The message was taken over but not sent due to an internal system or configuration error.
In this case as well, the message is tracked and made available for analysis or manual re-processing.

Ignored​

The message has been intentionally ignored and will not be processed.
This can occur, for example, when:

  • the message is outdated compared to another (see message correlation mechanisms), or
  • it has been manually marked as to be ignored by the user.

Operational Cycle Monitoring​

All message states are tracked and monitorable through the Flowlyze administration panel.
The interface enables you to:

  • view the list of messages in each state,
  • analyze error causes,
  • manually resend or ignore messages,
  • and consult overall delivery and throughput statistics.

Batch​

In the case of active message reading, for example during the import of a CSV file or reading records from a database table, the execution of the flow automatically generates a data batch.

The batch represents the set of imported messages during a single flow execution.
In other words, each time a flow is activated — manually or according to a schedule — Flowlyze collects all messages processed in that session and associates them with a unique batch.

Usage Example​

Typical use case

A typical case is that of a nightly flow scheduled for execution at midnight:
all messages read during that session are grouped into a single batch, enabling:

  • analyzing statistics per single execution (e.g., number of messages in Success or Error),

  • monitoring the process advancement status,

  • and verifying the overall import outcome.

Actions Linked to Execution​

Each batch execution can have automatic actions associated, configurable before or after flow startup.
These actions enable, for example:

  • sending email notifications at the end of processing,
  • generating and distributing statistical reports,
  • or executing custom post-processing procedures.

Thanks to this structured management, Flowlyze offers complete visibility on active flow execution, facilitating operational control and result analysis per single session.

Message Correlation​

For each Flow, it is possible to define a correlation criterion that enables identifying and managing related messages in an orderly manner.
Correlation is based on a unique identifier extracted from message content via a JsonPath selector, which points to a specific payload field (e.g., an order ID, product code, or document reference). Thanks to this mechanism, Flowlyze can recognize multiple versions of the same data and maintain only the most recent copy, avoiding duplications or unnecessary sends to the destination platform.

Correlation Purpose​

Correlation is particularly useful in cases where:

  • the same record can be received multiple times (e.g., subsequent updates of an order or a record),
  • it is necessary to ensure consistency of sent data,
  • and you want only the most updated version of the message to be processed and delivered.

Configuration​

The correlation mechanism is based on two main settings, configurable at the flow level:

  • Correlation Json Path
    A JsonPath format selector that identifies the unique identifier of the message from the source. Example: $.order.id to identify messages related to the same order.
  • Correlation Priority Path
    A JsonPath selector indicating the priority field to use for comparing multiple versions of the same message.
    This can be, for example, the last modification date or an incremental version number. Example: $.order.lastUpdated or $.version.

In summary, correlation enables Flowlyze to maintain logical data order and optimize the delivery process, ensuring that the destination always receives the most updated and consistent information available.