Results 1 to 10 of 10

Thread: Joining many files

  1. #1
    Contributor
    Join Date
    Mar 2010
    Location
    Melbourne
    Posts
    27

    Default Joining many files

    Hi there,

    I have 50 files, all with the same key to allow me to join them, but with different data held in them.

    I would like to create one table with all the data.

    Obviously I can join these one by one (49 times in total), but is there a faster way to join these that creating 49 look up nodes?

    Mike

  2. #2
    Contributor
    Join Date
    Feb 2009
    Location
    Chicago
    Posts
    194

    Default

    Hi Mike,

    Are they CSV files or similar?
    What you could do is the following:

    1. Place all the files in a directory and have them in the same format
    2. Add a Directory Node to look for the files.
    3. Pin the Output node of the Directory Node to a CSV Node. You may think that the CSV Node doesn't have an input but the it will allow it.

    You can use the same concept for Multiple BRD Files.

    That way all your data is joined. Assuming that all the fields/columns are the same just different data sets

    Wayne
    Last edited by xathras; 03-13-2013 at 06:47 PM.

  3. #3
    Contributor
    Join Date
    Mar 2010
    Location
    Melbourne
    Posts
    27

    Default

    Thanks Wayne but unfortunatel\y the challenge is that each file has different headers and a varying number of headers, depending on the file.

    I also have another potential challenge in the join in that, in some cases, different files have some (only some not all) columns that are the same.

    I expect to do this now is looking more like a manual process rather than automated.

    Mike

  4. #4
    Contributor
    Join Date
    Feb 2009
    Location
    Chicago
    Posts
    194

    Default

    Mike,

    Can you give me an example. See if i can come up with something for you.

    Wayne

  5. #5
    Lavastorm Employee stonysmith's Avatar
    Join Date
    Nov 2006
    Location
    Grapevine Tx
    Posts
    799

    Default

    I sent Mike a sample graph.. I'll work thru it with him and then post the solution here.

  6. #6

    Default Can you give more details?

    Can you share what the solution was?
    More specifically, how do you configure the Directory List Node to consolidate all the files listed in its Output in one single CSV file?
    Thanks!
    Quote Originally Posted by stonysmith View Post
    I sent Mike a sample graph.. I'll work thru it with him and then post the solution here.

  7. #7
    Lavastorm Employee stonysmith's Avatar
    Join Date
    Nov 2006
    Location
    Grapevine Tx
    Posts
    799

    Default

    If you simply add a directory list and then connect that to a Delimited File node, you can read multiple CSVs in a single pass. My solution mentioned above is unnecessary if all the CSVs have the same header.

  8. #8

    Default

    I am new to this application (will have formal training in 2 weeks) so I am not that familiar with configuring nodes.

    When I try what you suggest, I get an error message saying "Property ls.brain.node.convert,filename Expr not set"
    It seem to point to the fact that I don't have anything populated in the FilenameExpr box.

    Can you guide me on how I should populate the fields of the Delimited File node, using an example? I tried with the help guide but it did not really clarify what I need to do with FilenameExpr and other parameters.

    Thanks!


    Quote Originally Posted by stonysmith View Post
    If you simply add a directory list and then connect that to a Delimited File node, you can read multiple CSVs in a single pass. My solution mentioned above is unnecessary if all the CSVs have the same header.

  9. #9
    Lavastorm Employee
    Join Date
    Aug 2009
    Location
    Cologne
    Posts
    513

    Default

    Hi,

    Within the FilenameExpr you simply need to specify the name of the field in the input data which contains the filename to read.
    If this is coming from a Directory List node, then the field name should just be 'FileName'

    As a general note, if you are having difficulty understanding what a parameter is for, you can hover over the label for the parameter, and help will appear in a tool-tip informing you as to the purpose of the parameter.

    To understand more about the node you are trying to use, simply select the node and hit F1, or within the node editor, click "Node Help" at the top right of the dialog.

    Regards,
    Tim.

  10. #10

    Default

    That worked fine.
    Many thanks for the quick reply! Well Appreciated!

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •