Dataset components are generally used to read from/ write to serial/multi files. The basic dataset components are “input file”, “output file” ,”intermediate file” and “lookup file” “
There are number of dataset components but those two are mostly used dataset components are like “read multiple file” write multiple file” are used read from and write to more than one serial files.
Input File
Input File represents data records read as input to a graph from one or multiple serial files or from a multifile according to the dml specified.
If data for does not match with the dml error message ( metadata error) printed in the screen.
In the URL part of input file it is recommended to use variable ($ variable like $INPUT_FILEs)
We can use multiple files (of same type) as input
Click on partition radio and the click the edit button. In the edit box mention the variable name which points the files
And the variable has to be defined in a fnx file like
export INPUT_FILES=`ls -1 $AI_TEMP/MML/CCE*`
or in sandbox where the left column should have the variable name (INPUT_FILES ) and right column should have the definition ($AI_TEMP/MML/CCE*)
This INPUT_FILES points all the files under $AI_TEMP/MML directory which are stated with CCE.
In the read port of the input dml is required to be mentioned to read data from server and according to that dml specified. This dml can be embedded or path of the same can be mentioned.
Output File
Output File stores data records from a graph to one or multiple serial files or to a multifile according to the dml specified in the write port. The output file can be created in write or append mode or permission for the other user can be controlled .
Output File stores data records from a graph to one or multiple serial files or to a multifile according to the dml specified in the write port. The output file can be created in write or append mode or permission for the other user can be controlled .
Lookup File
Lookup File represents one or multiple serial files or a multifile of data records small enough to be held in main memory, letting a transform function retrieve records much more quickly than it could retrieve them if they were stored on disk.
Lookup File associates key values with corresponding data values to index records and retrieve them
Intermediate File
Intermediate File represents one or multiple serial files or a multifile of intermediate results that a graph writes during execution, and saves for your review after execution.
The upstream component writes to Intermediate File through Intermediate File’s write port. After the flow of data records into the write port is complete, the downstream component reads from Intermediate File’s read port. This guarantees that the writing and reading processes are in two separate phases.
There are number of dataset components but those two are mostly used dataset components are like “read multiple file” write multiple file” are used read from and write to more than one serial files.
Input File
Input File represents data records read as input to a graph from one or multiple serial files or from a multifile according to the dml specified.
If data for does not match with the dml error message ( metadata error) printed in the screen.
In the URL part of input file it is recommended to use variable ($ variable like $INPUT_FILEs)
We can use multiple files (of same type) as input
Click on partition radio and the click the edit button. In the edit box mention the variable name which points the files
And the variable has to be defined in a fnx file like
export INPUT_FILES=`ls -1 $AI_TEMP/MML/CCE*`
or in sandbox where the left column should have the variable name (INPUT_FILES ) and right column should have the definition ($AI_TEMP/MML/CCE*)
This INPUT_FILES points all the files under $AI_TEMP/MML directory which are stated with CCE.
In the read port of the input dml is required to be mentioned to read data from server and according to that dml specified. This dml can be embedded or path of the same can be mentioned.
Output File
Output File stores data records from a graph to one or multiple serial files or to a multifile according to the dml specified in the write port. The output file can be created in write or append mode or permission for the other user can be controlled .
Output File stores data records from a graph to one or multiple serial files or to a multifile according to the dml specified in the write port. The output file can be created in write or append mode or permission for the other user can be controlled .
Lookup File
Lookup File represents one or multiple serial files or a multifile of data records small enough to be held in main memory, letting a transform function retrieve records much more quickly than it could retrieve them if they were stored on disk.
Lookup File associates key values with corresponding data values to index records and retrieve them
Intermediate File
Intermediate File represents one or multiple serial files or a multifile of intermediate results that a graph writes during execution, and saves for your review after execution.
The upstream component writes to Intermediate File through Intermediate File’s write port. After the flow of data records into the write port is complete, the downstream component reads from Intermediate File’s read port. This guarantees that the writing and reading processes are in two separate phases.
