Results 1 to 6 of 6

Thread: HTTP Node - application sample

  1. #1

    Default HTTP Node - application sample

    Hi All

    My daily process involves getting data (a table) from centain website.

    Can someone show me how to set it up? For example, Can you please set up the ndoe so I can get the table from the following link:
    https://strikeprice.cftc.gov/StrikePriceFormat.aspx

    Thanks

  2. #2
    Lavastorm Employee
    Join Date
    Nov 2012
    Location
    Warrington, UK
    Posts
    200

    Default

    Hi jpstory,

    Wherever possible it is best to obtain data such as this in a standardized format such as XML or JSON data rather than web scraping the data from a HTML page as the latter approach is inherently brittle and will break if the format of the data on the page changes. However, with that caveat, here is a data flow that uses the HTTP node to get the raw page response data and then uses a Filter node and scripting to extract the data from the embedded table.

    HTTP_Node_Example_StrikePrice_Data.brg

    Regards,
    Adrian

  3. #3

    Default

    Hi aWiiliamns1024

    Thanks for your help, appreciate it!

    I got an error while trying to run your sample graph. The error says: "Error occurred during executing request: Unknown host: strikeprice.cftc.gov"

    As you suggested, it's better to obtain data in XML or JSON, can you also provide a sample graph using such method, thanks!

  4. #4
    Lavastorm Employee
    Join Date
    Nov 2012
    Location
    Warrington, UK
    Posts
    200

    Default

    The HTTP node worked for me.

    HTTP_Node_Input_Tab_configuration.png

    Are you using a proxy on your internet connection? If so you will need to configure the properties on the node's Proxy tab.

    As the website is using HTTPS you may need to install the SSL certificate. The SSL guide is located here for a default install of the Lavastorm client:

    C:\Program Files\Lavastorm\LAE6.1\docs\lal\SSL Connections Guide for Lavastorm.pdf

    The data is not available from that web site as XML or JSON data. I would recommend using the Contact Us link on the page to ask whether the CTFC can make the data set available in an open data format in line with other Open Data initiatives.

  5. #5
    Lavastorm Employee
    Join Date
    Nov 2012
    Location
    Warrington, UK
    Posts
    200

    Default

    You may want to try to isolate the issue by copying the HTTP node and configuring it to a HTTP site e.g. our home page: http://www.lavastorm.com/
    If the node downloads the data then it would point to the cause being SSL-related. If you cannot connect then it would imply a routing issue.

  6. #6
    Lavastorm Employee
    Join Date
    Nov 2012
    Location
    Warrington, UK
    Posts
    200

    Default

    Obviously, if the CTFC are not able to supply the data in XML/JSON format then CSV or Excel are alternatives, though not the preferred options.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •