Output Selected Columns into Multiple Files

[Eddie]

Hi,

I am trying to output one large concatenated file into several smaller individual files using the "Output Delimited (Multifile)" node, however I only want to output selected columns.

At the moment my concatenated data looks like this:

Header RowN, Unique IdentifierN, Unique IdentifierM
Header RowN+1, Unique IdentifierN+1, Unique IdentifierM+1
...
Header RowN+100, Unique IdentifierN+100, Unique IdentifierM+100
Data Row1, Unique IdentifierN, Unique IdentifierM
Data Row2, Unique IdentifierN, Unique IdentifierM
....
DataRow1488, Unique IdentifierN, Unique IdentifierM
Data Row1, Unique IdentifierN+1, Unique IdentifierM+1
Data Row2, Unique IdentifierN+1, Unique IdentifierM+1
....
DataRow1488, Unique IdentifierN+1, Unique IdentifierM+1
...
Data Row1, Unique IdentifierN+100, Unique IdentifierM+100
Data Row2, Unique IdentifierN+100, Unique IdentifierM+100
....
DataRow1488, Unique IdentifierN+100, Unique IdentifierM+100

In order to get the node to output the right header/data into each individual file, I have set up the "Output Delimited (Multifile)" node to output individual files based on the Unique Identifiers in Columns 2 and 3, however I don't actually want those columns data in the outputted file...

Is there any way to manipulate this node so that it only outputs selected columns, or is there another node which will remove the columns from the individual files?

Thanks,
Eddie

[Tim Meagher]

Hi,

Can you post the graph you are using, together with the dependent base libraries?

I can't seem to find the "Output Delimited (Multifile)" node you refer to in the core library. Is this something that someone at Meiridan (or a member of the Lavastorm services team) developed for your graphs?

Tim.

[Eddie]

Hi Tim,

Here is a copy of the graph, and the dependant base library.

Cheers,
Eddie

[Tim Meagher]

Hey,

Sorry I didn't realise that this was a part of the prototype library.
It doesn't look like there is any way to do this with the output delimited multifile node, unless you copy that node and modify the python code defined in the node code.

From the looks of it, it probably isn't too difficult to get the functionality you want, but it will involve a little bit of coding.

I'll add a note to the Lavastorm labs section about this request.

Tim.

[Tim Meagher]

Hi,

I looked again at the prototype library in Lavastorm labs, and while it looks like the library hasn't been updated, someone had already provided a version which allows you to exclude a field from the output.

It doesn't allow for multiple field exclusion, but I think generally you should be able to move all of the logic for determining the file to output into one field if necessary & then remove that field from the output.

I've attached that version of the node (including a fix that was also suggested in the same thread).

Tim.

[Eddie]

That works great

Thanks