Updates to the LAL library

[simon.wilby]

Hi all,

New in LAL 6.1.38 / LAL 6.0.38

• The Lavastorm Get for Salesforce node
• The HTTP node enhancement
• The K-means Advisor node enhancement
• The K-means Clustering node enhancement
• The R Library Package Download node enhancement

Details of New Functionality

The Lavastorm Get for Salesforce node queries SALESFORCE.COM objects via SOQL over the SALESFORCE.COM REST API. We recommend all new graphs use the “Lavastorm Get for Salesforce” node instead of the existing “Lavastorm Query for Salesforce” node which has been deprecated. The “Lavastorm Query for Salesforce” node will continue to operate for existing graphs.

The HTTP node has been enhanced to now include two “Truststore” parameters. These parameters enable the node to utilize a user specified Java Runtime Environment (JRE) trust store. LAL-4928

The K-Means Advisor node has been enhanced with a new “TransformData” parameter which provides the option to apply z-score standardization to each field in the input data. LAL-2306

The K-Means Clustering node has been enhanced with a new “TransformData” parameter which provides the option to apply z-score standardization to each field in the input data. LAL-2217

The R Library Package Download node has been amended to use our own repository https://lavastorm.exavault.com/p/LavastormStatRepo. This is due to compatibility issues with packages distributed on the official CRAN sites. Users of the R Library Package Download node do not need to do anything different. Users of the Power R node, who do not have the R Library Package node, are expected to use the above location in line with the install.packages command and save the libraries to <LAE temp directory>/TERR/libs/. The Lavastorm repository does not currently support linux distributions. The source files can be viewed here LAL-5379

Experimental Nodes
WARNING: A range of experimental nodes have been included with the LAE product that streamline a range of common data acquisition, preparation and analysis tasks, particularly for business users. When using nodes marked as experimental some features may not work. The name of the node or parameters may change, causing any graphs that use them to require updating. The documentation may be incomplete. Use of these nodes implies a willingness to accept these limitations.

• Change Fields - Changes the name and/or data type of selected fields in the input data. Selected fields are reordered
• Cleanse String Fields - Removes or replaces non-alphanumeric characters and reduces consecutive whitespace to a single space
• Compare Data Sets - Compares the values for a specified field in each of the configured (up to 3) data sources
• Compare Multiple Data Sets - Compares the values for a specified field in each of the configured data sources
• Complex Lookup - Provides complex lookups using logical operator-based rules
• CSV Data - Imports data from a CSV file or comma-separated data from an input field
• Date Range Filter - Filters data using a date range
• Date Range Lookup - Provides date-related lookups using logical operator-based rules
• Delimited Data - Parses delimited data from a specified file, a set of input files or from data fields in the input
• Dynamic Split String - Splits a string type field using a specified delimiter character
• Excel Options - Reads and converts a single-sheet Microsoft Excel (.xls, .xlsx) file for input
• Filter By Date Range - Filters data by data range
• Filter By Dates Within Time Period - Filters data to identify records that are within a specified time period of a reference date (or today)
• LAE Properties - Retrieves LAE properties information
• Load CSV File with 1 Additional Column - Reads and converts a CSV file that has an additional column
• Make R-Safe Fieldnames - Changes field names to valid R variable names
• More/Less/Equal Date Filter - Filters data by comparing the values in a date field agaist a specified date
• Multi-Format Date Conversion (with 2D checks) - Analyzes and normalizes dates in a field to have a consistent date format
• Multi-Format Date Conversion - Analyzes and normalizes dates in a field to have a consistent date format
• Multiple Way Data Profiling - Analyses the contents of up to four input data sets (e.g. database tables) to identify fields that may possibly contain primary keys and fields that contain foreign keys
• N Period Date Filter - Filters the data by checking the value in a date field to determine if it is within a specified 'window' time period of a specified date or today's date
• Oracle tnsnames.ora Analyzer - Extracts information from an Oracle tnsnames.ora file
• Power R Read Delimited File - Imports data from a delimited file
• Power R Read SAS Data File - Reads and converts a SAS Database (.sas7bdat) file for input
• Power R Stats Over Calendar Period - Calculates statistics for a time series over calendar periods
• Quick Acquisition - Imports data from a CSV or Excel file
• Rolling Statistics - Analyzes data to generate statistics for a rolling 'window' of records
• Select Fields - Filters the fields (columns) in the input data
• Split Field By Delimiters - Splits string data by delimiter into multiple columns
• Split String - Splits the contents a specified field using a given delimiter string
• String Based Lookup - Provides lookups on string variables using logical operator-based rules.
• Three Way Data Profiling - Analyses the contents of three input data sets (e.g. database tables) to identify fields that may possibly contain primary keys and fields that contain foreign keys
• Top/Bottom N Records - Extracts the top 'N' or bottom 'N' records from the data using the value of a specified field as the selection criterion
• Two Way Data Profiling - Analyses the contents of two input data sets (e.g. database tables) to identify fields that may possibly contain primary keys and fields that contain foreign keys

See the release notes for more details. the Release notes and download can be found on our download page.

Regards

Simon

[simon.wilby]

Hi all,

New in LAL 6.1.40 / LAL 6.0.40

Overview
New in Version 6.x.40 of LAL:

• The Predict Quantile Regression node (within the Statistical and Predictive Analytic Node Pack)
• The Lavastorm Get for Salesforce node enhancement
• The Lavastorm Query for Salesforce node (un-deprecated)

The Statistical and Predictive Analytic Node Pack (Powered by TIBCO®) is Premium licensed.
Note – The Predict Quantile Regression node is supported on the Windows platform, RedHat and SUSE Linux 64bit platform only.

Details of New Functionality

The Predict Quantile Regression node predicts the value of a dependent variable in a Quantile Regression model based on the value of the independent variables.
The Lavastorm Get for Salesforce node has been enhanced to return column names when the query returns no records. LAL-4726
The Lavastorm Query for Salesforce node has been un-deprecated; the node will be deprecated in a future release when all additional features are delivered in the Lavastorm Get for Salesforce node.

Bugfixes
The Send Email+ node now supports mixing the ToAddress and MessageBody parameters with literal or field options; previously the node would overwrite the MessageBody “Field” setting when the ToAddress was set to “literal”. LAL-4382

known issues/considerations
The Lavastorm Update for Salesforce node does not currently handle null entries for fields to be updated. LAL-5538

New in LAL 6.1.39 / LAL 6.0.39

Overview
New in Version 6.X.39 of LAL:

• The Lavastorm Metadata for Salesforce node
• The Lavastorm Update for Salesforce node
• The Quantile Regression node (within the Statistical and Predictive Analytic Node Pack)

The Statistical and Predictive Analytic Node Pack (Powered by TIBCO®) is Premium licensed.
Note – The Quantile Regression node is supported on the Windows platform, RedHat and SUSE Linux 64bit platform only.


Details of New Functionality

The Lavastorm Metadata for Salesforce node retrieves metadata in SALESFORCE.COM objects via the SALESFORCE.COM REST API.
The Lavastorm Update for Salesforce node inserts, updates, upserts or deletes records in SALESFORCE.COM objects via the SALESFORCE.COM REST API.
The Quantile Regression node models data using quantile regression allowing identification of data trends for conditional quantiles of a response variable distribution.

Bugfixes
The Excel File node default for the NoOutputPinErrorBehaviour parameter has been amended to log instead of Error LAL-5238 (LAL 6.1.39 only)
The Lavastorm Get for Salesforce node now supports queries for parent-to-child and child-to-parent relationships. LAL-5282
The Lavastorm Transact for Salesforce node requested a new session on each iteration, this resulted in BRE hanging if more than 10 instances were running. The new suite of Lavastorm for Salesforce nodes (Get, Metadata and Update) all output the session details so the same session can be used repeatedly. LAL-5077
The HTTP node now masks the TrustStoreFilePassword parameter LAL-5278


See the release notes for more details. the Release notes and download can be found on our download page.

Example graphs attached.

Regards

Simon

[simon.wilby]

Hi all,

New in LAL 6.1.41 / LAL 6.0.41

Overview

• The Cleanse String node enhancement
• The Data Converter node (Experimental)
• The Lavastorm Get / Metadata / Update for Salesforce node enhancement
• The Select Fields node enhancement
• The Spark Download node (Experimental)
• The Spark End Session node (Experimental)
• The Spark Java Submit node (Experimental)
• The Spark Sort node (Experimental)
• The Spark Start Session node (Experimental)
• The Spark Upload node (Experimental)

Details of New Functionality

The Cleanse String node has an improved performance. LAL-5643
The Data Converter node converts the data type of each field based on the discovered data type.
The Lavastorm Get / Metadata / Update for Salesforce nodes now support the use of a Proxy server. LAL-5556 / LAL-5557 / LAL-5558.
The Select Fields node has an improved performance. LAL-5644
The Spark Download node downloads data in the Avro file format from a Spark cluster.
The Spark End Session node ends a Spark session.
The Spark Java Submit node provides an interface with a Spark cluster to execute a Spark application.
The Spark Sort node sorts data in the Avro file format using a Spark cluster.
The Spark Start Session node starts a Spark session.
The Spark Upload node uploads data to a Spark cluster in the Avro file format.

Bugfixes
The Output Excel and Append Excel nodes now cater for dates greater than the year 2098. LAL-5563
The Change Metadata node now caters for dates greater than the year 2098. LAL-5507

Third party known issues/considerations

• The Avro 1.7.7 specification calls out some constraints on the metadata that it can support. Specifically it places restrictions on the names of fields:
  The Field names must:
  start with [A-Za-z_]
  subsequently contain only [A-Za-z0-9_]
• Avro 1.7.7 does not support date, time and datetime data types. As a result if you wish to upload data and use the Lavastorm nodes then these fields will need to be converted to string.

See the release notes for more details. the Release notes and download can be found on our download page.

Example graphs attached for the Spark based nodes. Please read the Spark Configuration v0.1.pdf before attempting to run the nodes.

Regards

Simon

[simon.wilby]

Hi all,

New in LAL 6.1.42 / LAL 6.0.42

Overview

• Tibco Enterprise Runtime for R (TERR) embedded R engine has been updated (6.1 only)
• The Spark Filter node (Experimental)
• The Spark Java Submit node enhancement
• The Spark Sample node (Experimental)

Details of New Functionality
The Tibco Enterprise Runtime for R (TERR) embedded R engine has been updated from 3.2 to 4.0 on both the Windows 64bit and Linux 64bit platforms. LAL-5550 (6.1 only)
The Spark Filter node filters data using a Spark cluster.
The Spark Java Submit node has been enhanced to allow customers to specify configuration properties for the Java application execution. LAL-5791
The Spark Sample node samples data using a Spark cluster.

Bugfixes
The Lavastorm Update for Salesforce node now advises which record has failed to update and stops executing at that record. LAL-5537
The Lavastorm Update for Salesforce node now completes updates where values are null. LAL-5538

See the release notes for more details. the Release notes and download can be found on our download page.

Example graphs attached for the Spark based nodes. Please read the Spark Configuration v0.1.pdf before attempting to run the nodes.


Regards

Simon

[simon.wilby]

Hi all,

New in LAL 6.1.43 / LAL 6.0.43

Overview

• The Lavastorm Get for Salesforce node enhancement
• The Spark Agg node (Experimental)
• The Spark Data Viewer node (Experimental)
• The Spark Join node (Experimental)
• Spark nodes enhancement

Details of New Functionality
The Lavastorm Get for Salesforce node now supports the SOQL convertCurrency and date functions. LAL-5871
The Spark Agg node aggregates data using a Spark cluster.
The Spark Data Viewer node previews data from Spark.
The Spark Join node joins data using a Spark cluster.
A number of Spark based nodes (i.e. those that include an InputSource parameter) have been enhanced to provide a default value for the InputSource parameter; this is so that users only need to connect the second output pin from a previous Spark node into the second input pin. LAL-5843

Bugfixes
The TIBCO Enterprise Runtime for R (TERR) based nodes (the Statistical and Predictive Analytic Node Pack) were failing to execute on the Windows 32bit operating system. The TERR remoteJvmArch property has been updated to specify the operating system bitness instead of the default 64bit option. LAL-5950 (LAL 6.1.43 only)

See the release notes for more details. the Release notes and download can be found on our download page.

Example graphs attached for the Spark based nodes. Please read the Spark Configuration v0.1.pdf before attempting to run the nodes.


Regards

Simon

[simon.wilby]

Hi all,

New in LAL 6.1.45 / LAL 6.0.45

Overview
Please Note: All of the Spark nodes now require Spark 1.5.0 as a minimum in the <LAE Install>/lib/java/lal/spark directory. In addition the Spark SQL Query node requires the Hive Client Configuration to be located on the laeServer machine. (See the Spark Configuration Guide for Lavastorm.pdf included in your installed documentation).

• The Archive Compress node
• The QVD File node
• The Spark SQL Query node (Experimental)
• The Spark Agg node enhancement
• The Spark Join node enhancement
• The Excel File node enhancements

Details of New Functionality
The Archive Compress node creates a compressed archive file containing one or more files. LAL-4961
The QVD File node imports data from a QVD file. LAL-3102. Additional software libraries are required in order for this node to work (commons-io-2.4.jar, irbi-fileutils-0.1.0.jar, jdbm-3.0-SNAPSHOT.jar, qvd-converter-core-1.5.0.jar and qvd-reader-1.5.0.jar).
Please contact Ralf Becher using the contact details below in order to purchase these libraries. Once you have those libraries they need to be saved to <LAE install Directory>/lib/java/lal.
IT Beratung Ralf Becher
Leipzig, Germany
ralf.becher@web.de

The Spark SQL Query node provides an interface with a Spark cluster to execute a Spark SQL query. LAL-6194
The Spark Agg node has new parameters included to allow customers to include or exclude fields from the aggregate function. LAL-5817
The Spark Join node has been enhanced to include new parameters so that users are able to filter on one or both datasets. LAL-6001
The Excel File node has a new parameter called IgnoreCellFormatting which allows the raw data underneath cell formatting to be imported to LAE. The default behavior is set to “Ignore” to allow for backwards compatibility. LAL-1554/gnats 5119

Bugfixes
The Trim Fields node now fully caters for fields with or without single quotes. LAL-5004
The Trim Fields node set to NULL if empty option has been improved to work on all fields. Also if the field names do not exist for the excluded/included option an exception is now returned. LAL-4049

Third party known issues/considerations
The Spark SQL Query node has highlighted some Apache issues with the following scenarios:

• Selecting a field with binary type fails with the exception “UnresolvedUnionException: Not in union ["bytes","null"]”. This is already raised on Apache JIRA: https://issues.apache.org/jira/browse/AVRO-1401
• Describe operation returns 3 fields (col_name, col_type, comment) however the comment field is handled as not “nullable” but returns a NULL value.

The Spark SQL Query node processes against Hive tables. When Hive tables are processed by the cluster, the minimum memory requirement is higher compared with the memory required to run other Spark nodes that do not access Hive. The DriverMemory and ExecutorMemory both have a minimum 5G threshold. Users are advised to increase this for larger environments.


New in LAL 6.1.44 / LAL 6.0.44

Overview
Please Note: All of the Spark nodes now require Spark 1.5.0 as a minimum in the <LAE Install>/lib/java/lal/spark directory (See the Spark Configuration Guide included in your installed documentation).

• The SharePoint 2010 suite of nodes enhancements
• The Spark Agg node enhancements
• The Spark Filter node enhancements
• The Spark Java Submit node enhancement
• The Spark Join node enhancement
• New Guides included

Details of New Functionality
The SharePoint 2010 suite of nodes have been enhanced to disable Forms Based Authentication. LAL-5739
The SharePoint 2010 suite of nodes have been enhanced to support situations where there is a URL-Folder Name mismatch. Please note: If there is a mismatch then additional processing is required to match up display names with underlying names used by the API. LAL-5623
The Spark Agg node has been enhanced to include the function name in the output columns. LAL-6112
The Spark Agg node now includes grouping functionality. Users are able to group aggregated data by one or more fields so that any records with identical values in the specified group fields are combined into a single record. LAL-5818
The Spark Filter node now allows a user to specify which fields are to be included or excluded on the output file(s). LAL-5814
The Spark Filter now includes the ability to filter on an additional two sets of filter criteria. LAL-5813
The Spark Java Submit node has a new parameter called SparkAssemblyJar. Users are now able to specify the JAR they wish the Spark application to execute with. Users are advised to review the ‘Spark Configuration Guide for Lavastorm.pdf’ located in their LAE installation. LAL-6075
The Spark Join node now allows the ability to select from an additional four join types (default join type ‘Inner’): Left Outer, Right Outer, Full Outer, Left Semi. LAL-5998
Three new documents have been added to the install of LAL. The documents included are: ‘CRAN Package Install Guide for Lavastorm.pdf’, ‘Spark Configuration Guide for Lavastorm.pdf’ and ‘SSL Connections Guide for Lavastorm.pdf’. LAL-6111
LAL V6.1.45 Release Notes © Lavastorm Analytics 2016 All Rights reserved Page 17

Known issues/considerations

• The Spark Sample node has been updated to return ‘N Records’ instead of ‘First N Records’. When using the 'N records' option, the order of the generated sample set is not guaranteed to reflect the order of the data in the input file(s).

Attached are example graphs for the Archive Compress and Spark SQL nodes

Regards

Simon

[awilliams1024]

Hi All,

LAL 6.1.48 (Compatible with LAE 6.1.x) is now available.

New in LAL 6.1.48

Overview

• Data Converter node GA
• Spark Agg node enhancement
• Spark nodes Kerberos keytab authentication enhancement
• Spark Download and Spark Data Viewer string data type enhancement
• Spark nodes delimited file support enhancements
• Spark nodes GA

Details of New Functionality
The status of the (previously Experimental) Data Converter node has now been changed from experimental to fully supported.

The Spark Agg node now outputs a GroupBy field when the field is also included in an aggregation function. LAL-6480

The Spark nodes have been enhanced to offer an additional Kerberos authentication method which uses a secure keytab file.

The Spark Download node and Spark Data Viewer node have been enhanced to enable the customer to specify whether string fields should be output as unicode field type (default) or as string field type. LAL-6870
A number of Spark nodes have been enhanced to allow customers to specify a delimited file as the node’s input data source instead of the default Avro file format. Configuration options enable the customer to specify: the delimiter character, whether the delimited file has a header record, the quote character, the escape character and whether empty strings are to be considered NULL values. By default, the nodes attempt to automatically identify the schema. The schema can also be explicitly defined. LAL-6805, LAL-6697

The status of the (previously Experimental) Spark nodes has now been changed from experimental to fully supported. This relates to the following nodes: Spark Start Session, Spark End Session, Spark Upload, Spark Download, Spark Data Viewer, Spark Agg, Spark Filter, Spark Join, Spark Sample, Spark Sort, Spark SQL Query.

Bugfixes
Some extraneous parameters are no longer visible on the JSON Data node and the Logistic Regression Diagnostics node. LAL-6523, LAL-6840
Duplicate help text was removed from the Spark SQL Query node’s help information. LAL-6797

Removed Functionality
The following experimental nodes have been removed from the installation:

• Complex Lookup
• Date Range Lookup
• String Based Lookup

Third party known issues/considerations
In the situation where a Spark node is configured to import data from a delimited file and the schema has been defined by the user, the schema must be valid. When the schema is invalid the node execution may complete successfully but no records will be output.

The Release Notes and installers can be found on our download page.

Regards,
Adrian

[sobhsepid]

X-Ref MetaData node added - takes 2 inputs and performs a cross-reference against the metadata of the inputs, to find the common fields and types of the two inputs.