Results 1 to 6 of 6

Thread: Output Selected Columns into Multiple Files

  1. #1

    Default Output Selected Columns into Multiple Files

    Hi,

    I am trying to output one large concatenated file into several smaller individual files using the "Output Delimited (Multifile)" node, however I only want to output selected columns.

    At the moment my concatenated data looks like this:

    Header RowN, Unique IdentifierN, Unique IdentifierM
    Header RowN+1, Unique IdentifierN+1, Unique IdentifierM+1
    ...
    Header RowN+100, Unique IdentifierN+100, Unique IdentifierM+100
    Data Row1, Unique IdentifierN, Unique IdentifierM
    Data Row2, Unique IdentifierN, Unique IdentifierM
    ....
    DataRow1488, Unique IdentifierN, Unique IdentifierM
    Data Row1, Unique IdentifierN+1, Unique IdentifierM+1
    Data Row2, Unique IdentifierN+1, Unique IdentifierM+1
    ....
    DataRow1488, Unique IdentifierN+1, Unique IdentifierM+1
    ...
    Data Row1, Unique IdentifierN+100, Unique IdentifierM+100
    Data Row2, Unique IdentifierN+100, Unique IdentifierM+100
    ....
    DataRow1488, Unique IdentifierN+100, Unique IdentifierM+100

    In order to get the node to output the right header/data into each individual file, I have set up the "Output Delimited (Multifile)" node to output individual files based on the Unique Identifiers in Columns 2 and 3, however I don't actually want those columns data in the outputted file...

    Is there any way to manipulate this node so that it only outputs selected columns, or is there another node which will remove the columns from the individual files?

    Thanks,
    Eddie

  2. #2
    Lavastorm Employee
    Join Date
    Aug 2009
    Location
    Cologne
    Posts
    513

    Default

    Hi,

    Can you post the graph you are using, together with the dependent base libraries?

    I can't seem to find the "Output Delimited (Multifile)" node you refer to in the core library. Is this something that someone at Meiridan (or a member of the Lavastorm services team) developed for your graphs?

    Tim.
    Last edited by Tim Meagher; 01-24-2012 at 10:34 AM. Reason: Referenced Vodafone not Meridian, sorry ... was responding to two posts at once & got confused...

  3. #3

    Default Lavastorm Graph

    Hi Tim,

    Here is a copy of the graph, and the dependant base library.

    Cheers,
    Eddie
    Attached Files Attached Files

  4. #4
    Lavastorm Employee
    Join Date
    Aug 2009
    Location
    Cologne
    Posts
    513

    Default

    Hey,

    Sorry I didn't realise that this was a part of the prototype library.
    It doesn't look like there is any way to do this with the output delimited multifile node, unless you copy that node and modify the python code defined in the node code.

    From the looks of it, it probably isn't too difficult to get the functionality you want, but it will involve a little bit of coding.

    I'll add a note to the Lavastorm labs section about this request.

    Tim.

  5. #5
    Lavastorm Employee
    Join Date
    Aug 2009
    Location
    Cologne
    Posts
    513

    Default

    Hi,

    I looked again at the prototype library in Lavastorm labs, and while it looks like the library hasn't been updated, someone had already provided a version which allows you to exclude a field from the output.

    It doesn't allow for multiple field exclusion, but I think generally you should be able to move all of the logic for determining the file to output into one field if necessary & then remove that field from the output.

    I've attached that version of the node (including a fix that was also suggested in the same thread).

    Tim.
    Attached Files Attached Files

  6. #6

    Default

    That works great

    Thanks

Similar Threads

  1. Unzip .tar.gz files and Read XML files
    By credoblado in forum Nodes
    Replies: 3
    Last Post: 11-20-2009, 05:26 AM
  2. Output to multiple CSV files
    By PThai in forum Nodes
    Replies: 1
    Last Post: 09-17-2009, 03:49 AM
  3. Multiple BRD files and HeaderLess Entries
    By phourigan_MDS in forum Data Acquisition
    Replies: 2
    Last Post: 07-13-2009, 09:11 PM
  4. Naming Output Files Using Data in Graph
    By Jonser1246 in forum Nodes
    Replies: 10
    Last Post: 10-18-2008, 01:50 AM
  5. Concatenating multiple files
    By roeperg in forum Data Acquisition
    Replies: 3
    Last Post: 06-11-2008, 08:45 PM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •