A flow carries a stream of data between components in a graph. Flows connect components via ports. Ab Initio software supplies four kinds of flows with different patterns:
• straight
• fan-in
• fan-out
• all-to-all
Straight
Straight flows connect components that have the same number of partitions. Partitions of components connected with straight flows have a one-to-one correspondence. Straight flows are the most common flow pattern
Fan-in
Fan-in flows connect components with a large number of partitions to components with a smaller number of partitions. The most common use of fan-in is to connect flows to Departition components. This flow pattern is used to merge data divided into many segments back into a single segment, so other programs can access the data.
When you connect a component running in parallel to any component via a fan-in flow, the number of partitions of the original component must be a multiple of the number of partitions of the component receiving the fan-in flow. For example, you can connect a component running 9 ways parallel to a component running 3 ways parallel, but not to a component running 4 ways parallel.
fan-out
Fan-out flows connect components with a small number of partitions to components with a larger number of partitions. The most common use of fan-out is to connect flows from partition components. This flow pattern is used to divide data into many segments for performance improvements.
When you connect a Partition component running in parallel to another component running in parallel via a fan-out flow, the number of partitions of the component receiving the fan-out flow must be a multiple of the number of partitions of the Partition component. For example, you can connect a Partition component with 3 partitions via a fan-out flow to a component with 9 partitions, but not to a component with 10 partitions.
To deal with the latter case, Repartition the data by inserting a Departition component after the Partition component. This turns the fan-out flow into an All-to-all flow , allowing a record from any partition of the Partition component to flow into any partition of the target component.
all-to-all
All-to-all flows typically connect components with different numbers of partitions. Data from any of the upstream partitions is sent to any of the downstream partitions. The most common use of all-to-all flows is to repartition data as in the following example
• straight
• fan-in
• fan-out
• all-to-all
Straight
Straight flows connect components that have the same number of partitions. Partitions of components connected with straight flows have a one-to-one correspondence. Straight flows are the most common flow pattern
Fan-in
Fan-in flows connect components with a large number of partitions to components with a smaller number of partitions. The most common use of fan-in is to connect flows to Departition components. This flow pattern is used to merge data divided into many segments back into a single segment, so other programs can access the data.
When you connect a component running in parallel to any component via a fan-in flow, the number of partitions of the original component must be a multiple of the number of partitions of the component receiving the fan-in flow. For example, you can connect a component running 9 ways parallel to a component running 3 ways parallel, but not to a component running 4 ways parallel.
fan-out
Fan-out flows connect components with a small number of partitions to components with a larger number of partitions. The most common use of fan-out is to connect flows from partition components. This flow pattern is used to divide data into many segments for performance improvements.
When you connect a Partition component running in parallel to another component running in parallel via a fan-out flow, the number of partitions of the component receiving the fan-out flow must be a multiple of the number of partitions of the Partition component. For example, you can connect a Partition component with 3 partitions via a fan-out flow to a component with 9 partitions, but not to a component with 10 partitions.
To deal with the latter case, Repartition the data by inserting a Departition component after the Partition component. This turns the fan-out flow into an All-to-all flow , allowing a record from any partition of the Partition component to flow into any partition of the target component.
all-to-all
All-to-all flows typically connect components with different numbers of partitions. Data from any of the upstream partitions is sent to any of the downstream partitions. The most common use of all-to-all flows is to repartition data as in the following example
