Results 1 to 5 of 5

Thread: Excluding fields by Wildcard and Pattern

  1. #1
    Larry Tolleson ltolleson's Avatar
    Join Date
    Nov 2006
    Location
    Dallas-Fort Worth
    Posts
    189

    Lightbulb Excluding fields by Wildcard and Pattern

    Many times we find there are numerous fields in our graphs that need to be removed from the data stream. Normally we would have to exclude those files one-by-one. There are two soon to be documented features in BrainScript that now allow fields to be excluded from the output by using either wildcard (*) or pattern (regexp) values. The attached graph gives examples of each of these new functions and how they can be used in your graphs.

    If you have any questions or comments about this functionality please reply to this thread.
    Attached Files Attached Files
    Last edited by ltolleson; 09-22-2009 at 03:00 PM.

  2. #2
    Larry Tolleson ltolleson's Avatar
    Join Date
    Nov 2006
    Location
    Dallas-Fort Worth
    Posts
    189

    Default Update

    I just realized that the wildcard and pattern functions work with the emit statement as well.

    Example

    emit wildcard "ACCT_*"


    This example says output all fields that start with the string "ACCT_"

  3. #3
    New Contributor
    Join Date
    Oct 2009
    Location
    Atlanta
    Posts
    3

    Default Use wildcard to handle GST/HST field name

    I have several hundred CSV files containing a column heading GST/HST.
    The filter node can't handle the slash.
    Your post really helped because I could not find how to handle this in the BRAIN Script Help or the filter node help.

    I used the wildcard to get around this issue.

    emit wildcard "GST*"

    The field comes over as GST/HST and then I must use single quotes 'GST/HST' to use the field in other nodes.

    Anybody with a better way please let me know.

  4. #4

    Default Use wildcard to handle GST/HST field name

    What version of BRE are you using? With 4.1.5 I was unable to reproduce the issue you described. I used a Static Data node and changed the string column "color" to "GST/HST", ran it through a Filter node with:

    emit 'GST/HST'

    and it worked fine. I then output the Static Data node to a .csv file (Output Delimited node), input using the CSV File node, and ran through the same filter and again it worked.

    It appears that there may have been some confusion over the use of the single quotes around the field (column) name. If the field name is alphanumeric and unambiguous, it can be used without the single quotes (a.k.a. "tics"). Also a field name can be referenced without single quotes if it consists of an identifier, or a list of identifiers connected by "." . Field names can be prefixed with input#: without single quotes as well.

    The 'GST/HST' field name contains the front slash, which is a special character denoting a command option. It needs to be interpreted literally and so must be "escaped" somehow. The single quotes do this.

    Attached is the graph I used to check some things. Perhaps it might provide some clariifcation.

    Cheers,

    - Bob
    Attached Files Attached Files

  5. #5
    New Contributor
    Join Date
    Oct 2009
    Location
    Atlanta
    Posts
    3

    Default

    I am using BRE v4.1.5.0 Build 268.

    I think I caused the problem myself in my CSV File input node by specifying the column name GST/HST in the list of field names. I could have just specified a column heading that didn't include escape characters.

    Here is my original code just in case you want to recreate the same situation.

    My original CSV Input node.

    node:CSV_File
    bretype:core::CSV File
    editor:sortkey=4b6993d031166bf4_2
    input:4b6993d010a7405c/=Get_Billing_files__Jan.40fd2c747c2b1c0a
    output:@41e6c6cd11e613c6/=
    prop:FieldNames=Location,Province,Terminal,Card,Se rial #,Date,Time,Status,Reference #,Unit Cost,# of Cards,Gross,Discount %,Net,GST/HST,Total Due,Customer
    prop:FilenameExpr=<<EOX
    FileName
    EOX
    prop:QuotedDoubleQuotes=Literal Quote if no EscapeCharacters
    editor:XY=140,60
    end:CSV_File



    My Filter node using the wildcard
    node:Filter_15
    bretype:core::Filter
    editor:sortkey=4b73174d555b2164_13
    input:@40fd2c74167f1ca2/=CSV_File.41e6c6cd11e613c6
    output:@40fd2c7420761db6/=
    prop:Script=<<EOX
    UseThisFieldToAggregate = "Agg"
    emit Card.trim() as Card
    emit 'Serial #'.trim() as SerialNumber
    emit Date.trim() as Date
    emit Time.trim() as Time
    emit Status.trim() as Status
    emit 'Reference #'.trim() as ReferenceNumber
    emit 'Unit Cost'.trim() as UnitCost
    emit '# of Cards'.trim() as NumberOfCards
    emit Gross.trim() as Gross
    emit 'Discount %'.trim() as DiscountPercent
    emit Net.trim() as Net
    emit wildcard "GST*"
    emit 'Total Due'.trim() as TotalDue
    emit UseThisFieldToAggregate
    EOX
    editor:XY=260,60
    end:Filter_15

Similar Threads

  1. Excluding columns from output
    By stunelson in forum Nodes
    Replies: 12
    Last Post: 08-07-2018, 01:20 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •