Results 1 to 10 of 10

Thread: Import data from a Tableau TDE file

  1. #1

    Default Import data from a Tableau TDE file

    Hi, We've just started using Tableau as our front end and are now using the "output Tableau" TDE publish node. Is there any way to import a TDE ?

    We'd like to create the TDE and then append when new data becomes available, which would require reading the TDE back in to LS then overwriting the TDE file with the updated data.

    Thanks,
    Dave

  2. #2
    Lavastorm Employee
    Join Date
    Nov 2012
    Location
    Warrington, UK
    Posts
    230

    Default

    Hi Dave,

    the Tableau Data Extract (TDE) file format is Tableau proprietary. Looking online, there have been a number of requests for Tableau to open source the TDE formats so that other applications can read extract files. However, there is a statement on the Tableau community website that says: "The existing TDE API will continue to create TDE extracts but we have no plan to open source it."

    The post mentions that the new Tableau Hyper extract will replace the existing Tableau Data Engine and that "there will be an API for creating and updating Hyper extracts". No details of the API are available at this time.

    Regards,
    Adrian

  3. #3
    Lavastorm Employee gmullin's Avatar
    Join Date
    May 2014
    Location
    Chicago
    Posts
    162

    Default

    If you can't easily read the TDE file an easy solution would be to write/append the data to a BRD file at the same time you write the TDE file. Then you can easily read back in whatever you had in TDE by using that BRD file.

  4. #4

    Default

    Thanks all, curious how Lavastorm can create a TDE but not read one ?

    Also the second issue underlying this is that the data sets I'm working with are massive. Using BRD or CSV the file is over 5GB, but LS compresses this down to 100mb as a TDE. I'm trying desperately to balance the required work with a workable storage solution, and I can't shrink the underlying data any further.

    So I guess a secondary question is, if we can't read from a TDE, is there any other format I can use that heavily compresses the data ? XLSX won't work, the node just doesn't run even if I adjust the JvmMaxHeapSize, and I would prefer it in a single file.

    Data Warehousing sadly is not an option.

  5. #5
    Lavastorm Employee stonysmith's Avatar
    Join Date
    Nov 2006
    Location
    Grapevine Tx
    Posts
    781

    Default

    Quote Originally Posted by dglavastorm View Post
    Thanks all, curious how Lavastorm can create a TDE but not read one ?
    We use a library of code provided by Tableau - that library can write a TDE, but can't read one.
    In simple terms, Tableau themselves haven't provided a mechanism for reading, and their format is proprietary - they have not yet provided information sufficient for other companies to read the TDE files.

  6. #6

    Default

    Makes sense, thanks.

    So are there any formats the LS can write to that offer good compression ? And I don't mean compress to an archive like ZIP, it needs to be readable.

    BRD didn't really compress, it just made reading the data back in faster.

  7. #7
    Lavastorm Employee gmullin's Avatar
    Join Date
    May 2014
    Location
    Chicago
    Posts
    162

    Default

    I think the answer to your question is no. I tried to write out a BRD file of around 50mb that as TDE was 50kb but any other format I tried the file was of similar size if not larger. The only other option is to compress the file, but you said that's not an option for you. Maybe somebody has a custom node out there? I'd be happy to be proven wrong.

  8. #8

    Default

    Thanks, that unfortunately matches everything I've tested as well, including using a Teradata datalab. Whatever Tableau is doing, their compression is sensational.

  9. #9

    Default

    Are there plans to upgrade the Output Tableau node to support the new hyper format? Thanks.

  10. #10
    Lavastorm Employee
    Join Date
    Nov 2012
    Location
    Warrington, UK
    Posts
    230

    Default

    Hi,
    we have no plans to provide support for the Tableau .hyper extract format on the Lavastorm product.
    However, this may be something that could be considered for the Dataverse product.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •