Page 1 of 2 12 LastLast
Results 1 to 10 of 12

Thread: File import via HTTP transfer

  1. #1
    New Contributor
    Join Date
    Sep 2011
    Location
    Bellevue, WA
    Posts
    4

    Post File import via HTTP transfer

    Does anyone out there have any nodes developed to do directory and file transfer for HTTP?

    I have a number of .csv files posted on a server accessible via http. Currently I navigate via browser to get a directory listing and then click on each individual file and "save as" and then use the standard directory and csv nodes to get them into my graph for analysis.

    The ideal solution would be to have a HTTP directory node and HTTP file transfer node so that I don't have to go through the recursive manual process of "save as".

  2. #2
    Contributor
    Join Date
    Aug 2011
    Location
    Helsinki, Finland
    Posts
    16

    Default

    Could you just use wget to download files to your computer first? You could even automate the process using Scheduled Tasks -feature in Windows (Control Panel).

    Code:
    wget -nd -r --accept=csv,txt -l 1 www.xyz.org/datadir

  3. #3
    Lavastorm Employee stonysmith's Avatar
    Join Date
    Nov 2006
    Location
    Grapevine Tx
    Posts
    799

    Default

    Attached is a node that I wrote some time ago. You'll need to change how rows 31-36 process the data if you are reading a binary file such as an Excel document.
    Attached Files Attached Files

  4. #4
    Contributor
    Join Date
    Aug 2011
    Location
    Helsinki, Finland
    Posts
    16

    Default

    Quote Originally Posted by stonysmith View Post
    Attached is a node that I wrote some time ago. You'll need to change how rows 31-36 process the data if you are reading a binary file such as an Excel document.
    Thanks, the node works great!

    I have one further question though: Would it be possible to fetch XML data with the node and pass data straight to XMLpy-node somehow? I'm aiming to parse some 100k exchange rates from http://www.ecb.europa.eu/stats/eurof...fxref-hist.xml and would find XMLpy useful here.

  5. #5
    Lavastorm Employee
    Join Date
    Sep 2009
    Location
    Boston, Massachusetts, USA
    Posts
    50

    Default

    Dear Erno,

    All Acquisition nodes are built to bring a file into the LAE environment from the file system. Because of this, there is no way to pass XML data into the XMLpy File node via an input BRD. Instead, the Read WebPage node can write the file to disk and then output the full path of the file it created to its output pin. The XMLpy File node is built to take an input filename from an input BRD via the FilenameExpr parameter.

    Feel free to write back if you have any difficulties.

    Rocco

  6. #6
    Contributor
    Join Date
    Aug 2011
    Location
    Helsinki, Finland
    Posts
    16

    Default

    Quote Originally Posted by rpigneri View Post
    Dear Erno,

    All Acquisition nodes are built to bring a file into the LAE environment from the file system. Because of this, there is no way to pass XML data into the XMLpy File node via an input BRD. Instead, the Read WebPage node can write the file to disk and then output the full path of the file it created to its output pin. The XMLpy File node is built to take an input filename from an input BRD via the FilenameExpr parameter.

    Feel free to write back if you have any difficulties.

    Rocco
    Could you give me an example how to write to a file directly from Read WebPage node? With BRD output I'm able to get almost perfect results, but it adds htmlpage:string -line to the beginning of the ouput, which messes XMLpy.
    Attached Files Attached Files

  7. #7
    Lavastorm Employee stonysmith's Avatar
    Join Date
    Nov 2006
    Location
    Grapevine Tx
    Posts
    799

    Default

    Quote Originally Posted by erno View Post
    Could you give me an example how to write to a file directly from Read WebPage node? With BRD output I'm able to get almost perfect results, but it adds htmlpage:string -line to the beginning of the ouput, which messes XMLpy.
    Attached is a graph that shows how to use OutputDelimited to write the file.

    To get OuptutDelimted to work:
    - specify FieldNames=None
    - specify some value for QuoteCharacter

    The tricky part here is to get OutputDelimited to work right, you have to specify a QuoteCharacter that does not exist in the input data. It might take some experimenting to find the right character to use. In this case we used the backslash, but we also got it to work with .

    I would be interested to see what you use for the XMLpy node. I'm not very familiar with it, and I'm not sure how it should be configured to read this particular XML.
    Attached Files Attached Files

  8. #8
    Contributor
    Join Date
    Aug 2011
    Location
    Helsinki, Finland
    Posts
    16

    Default

    I feel awfully stupid for not figuring this out before. Thank you so much, got my exchange rate fetch working!

  9. #9

    Default

    Dear All

    I tried to use the attached euro.brg sample file, but I received :
    python errno 10061 no connection could be made because the target machine actively refused it

    What can be the problem ?

    Best regards

    Zoltan

  10. #10
    Lavastorm Employee
    Join Date
    Aug 2009
    Location
    Cologne
    Posts
    513

    Default

    Hi,

    I'm not sure about that specific node or graph, because it was just posted here on the forums without any Lavastorm support.
    However, since then, the HTTP node has been released with LAL.
    So if you are trying to make a HTTP request, then you should just be able to get the latest LAL release and use the HTTP node with the parameters set as shown in the attached example BRG.

    Then, you can either just save off the XML using an output BRD, or pass it through the Convert Tabular XML node in order to have the data fields returned converted to standard LAE data fields rather than an XML response chunk.

    Tim.
    Attached Files Attached Files

Similar Threads

  1. Replies: 4
    Last Post: 11-14-2013, 04:51 PM
  2. Exclude rows and then import
    By mikewillett in forum Data Acquisition
    Replies: 4
    Last Post: 11-23-2011, 12:04 AM
  3. CSV import
    By MarkCunningham in forum Nodes
    Replies: 4
    Last Post: 07-19-2011, 09:10 AM
  4. Import file that as blank fields
    By joanaramada in forum Nodes
    Replies: 1
    Last Post: 05-10-2011, 02:22 PM
  5. Opening a file where directory and file name change
    By TheBishop in forum Data Acquisition
    Replies: 3
    Last Post: 06-10-2010, 03:54 AM

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •