Results 1 to 2 of 2

Thread: Sort node performance varies wildly on same data set

Hybrid View

  1. #1

    Default Sort node performance varies wildly on same data set

    Hi, just trying to understand the behaviour we're seeing on the sort node.
    This is desktop LS v6, run on the same PC with no other work being run at the same time.
    Dataset is 34m rows with 7 columns, sorting on 2 columns - a 10 character string and a datetime.

    A week ago when I ran this node, after an hour and a half it still wasn't completed. It didn't error, just kept running, forcing me to sort the data external to Lavastorm and re-import.

    Yesterday it completed in 22 mins so I thought it was just an error last time.

    However today it's been running for over an hour 10 mins again and still running.

    The source data is exactly the same in each case, and again the PC is only being used for this one purpose with nothing else running as verified in Task Manager.

    What could be driving this wildly different performance, and what optimisation steps could I do ? The 10 character string is actually all numbers, so not sure if changing it to int() or something would make any difference.

    Thanks,
    Dave

  2. #2

    Default

    I've had this happen but I thought nothing of it. I just let it run to completion. Maybe someone else can shed some light on this. I'm interested to find out if something's wrong too.

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •