LAL 6.1.38 (Compatible with LAE 6.1.x) and LAL 6.0.38 (Compatible with LAE 6.0.x)
Hi all,
New in LAL 6.1.38 / LAL 6.0.38
The Lavastorm Get for Salesforce node
The HTTP node enhancement
The K-means Advisor node enhancement
The K-means Clustering node enhancement
The R Library Package Download node enhancement
Details of New Functionality
The Lavastorm Get for Salesforce node queries SALESFORCE.COM objects via SOQL over the SALESFORCE.COM REST API. We recommend all new graphs use the “Lavastorm Get for Salesforce” node instead of the existing “Lavastorm Query for Salesforce” node which has been deprecated. The “Lavastorm Query for Salesforce” node will continue to operate for existing graphs.
The HTTP node has been enhanced to now include two “Truststore” parameters. These parameters enable the node to utilize a user specified Java Runtime Environment (JRE) trust store. LAL-4928
The K-Means Advisor node has been enhanced with a new “TransformData” parameter which provides the option to apply z-score standardization to each field in the input data. LAL-2306
The K-Means Clustering node has been enhanced with a new “TransformData” parameter which provides the option to apply z-score standardization to each field in the input data. LAL-2217
The R Library Package Download node has been amended to use our own repository https://lavastorm.exavault.com/p/LavastormStatRepo. This is due to compatibility issues with packages distributed on the official CRAN sites. Users of the R Library Package Download node do not need to do anything different. Users of the Power R node, who do not have the R Library Package node, are expected to use the above location in line with the install.packages command and save the libraries to <LAE temp directory>/TERR/libs/. The Lavastorm repository does not currently support linux distributions. The source files can be viewed here LAL-5379
Experimental Nodes
WARNING: A range of experimental nodes have been included with the LAE product that streamline a range of common data acquisition, preparation and analysis tasks, particularly for business users. When using nodes marked as experimental some features may not work. The name of the node or parameters may change, causing any graphs that use them to require updating. The documentation may be incomplete. Use of these nodes implies a willingness to accept these limitations.
Change Fields - Changes the name and/or data type of selected fields in the input data. Selected fields are reordered
Cleanse String Fields - Removes or replaces non-alphanumeric characters and reduces consecutive whitespace to a single space
Compare Data Sets - Compares the values for a specified field in each of the configured (up to 3) data sources
Compare Multiple Data Sets - Compares the values for a specified field in each of the configured data sources
Complex Lookup - Provides complex lookups using logical operator-based rules
CSV Data - Imports data from a CSV file or comma-separated data from an input field
Date Range Filter - Filters data using a date range
Date Range Lookup - Provides date-related lookups using logical operator-based rules
Delimited Data - Parses delimited data from a specified file, a set of input files or from data fields in the input
Dynamic Split String - Splits a string type field using a specified delimiter character
Excel Options - Reads and converts a single-sheet Microsoft Excel (.xls, .xlsx) file for input
Filter By Date Range - Filters data by data range
Filter By Dates Within Time Period - Filters data to identify records that are within a specified time period of a reference date (or today)
LAE Properties - Retrieves LAE properties information
Load CSV File with 1 Additional Column - Reads and converts a CSV file that has an additional column
Make R-Safe Fieldnames - Changes field names to valid R variable names
More/Less/Equal Date Filter - Filters data by comparing the values in a date field agaist a specified date
Multi-Format Date Conversion (with 2D checks) - Analyzes and normalizes dates in a field to have a consistent date format
Multi-Format Date Conversion - Analyzes and normalizes dates in a field to have a consistent date format
Multiple Way Data Profiling - Analyses the contents of up to four input data sets (e.g. database tables) to identify fields that may possibly contain primary keys and fields that contain foreign keys
N Period Date Filter - Filters the data by checking the value in a date field to determine if it is within a specified 'window' time period of a specified date or today's date
Oracle tnsnames.ora Analyzer - Extracts information from an Oracle tnsnames.ora file
Power R Read Delimited File - Imports data from a delimited file
Power R Read SAS Data File - Reads and converts a SAS Database (.sas7bdat) file for input
Power R Stats Over Calendar Period - Calculates statistics for a time series over calendar periods
Quick Acquisition - Imports data from a CSV or Excel file
Rolling Statistics - Analyzes data to generate statistics for a rolling 'window' of records
Select Fields - Filters the fields (columns) in the input data
Split Field By Delimiters - Splits string data by delimiter into multiple columns
Split String - Splits the contents a specified field using a given delimiter string
String Based Lookup - Provides lookups on string variables using logical operator-based rules.
Three Way Data Profiling - Analyses the contents of three input data sets (e.g. database tables) to identify fields that may possibly contain primary keys and fields that contain foreign keys
Top/Bottom N Records - Extracts the top 'N' or bottom 'N' records from the data using the value of a specified field as the selection criterion
Two Way Data Profiling - Analyses the contents of two input data sets (e.g. database tables) to identify fields that may possibly contain primary keys and fields that contain foreign keys
See the release notes for more details. the Release notes and download can be found on our download page.
LAL 6.1.40 (Compatible with LAE 6.1.x) and LAL 6.0.40 (Compatible with LAE 6.0.x)
Hi all,
New in LAL 6.1.40 / LAL 6.0.40
Overview
New in Version 6.x.40 of LAL:
The Predict Quantile Regression node (within the Statistical and Predictive Analytic Node Pack)
The Lavastorm Get for Salesforce node enhancement
The Lavastorm Query for Salesforce node (un-deprecated)
The Statistical and Predictive Analytic Node Pack (Powered by TIBCO®) is Premium licensed.
Note – The Predict Quantile Regression node is supported on the Windows platform, RedHat and SUSE Linux 64bit platform only.
Details of New Functionality
The Predict Quantile Regression node predicts the value of a dependent variable in a Quantile Regression model based on the value of the independent variables.
The Lavastorm Get for Salesforce node has been enhanced to return column names when the query returns no records. LAL-4726
The Lavastorm Query for Salesforce node has been un-deprecated; the node will be deprecated in a future release when all additional features are delivered in the Lavastorm Get for Salesforce node.
Bugfixes
The Send Email+ node now supports mixing the ToAddress and MessageBody parameters with literal or field options; previously the node would overwrite the MessageBody “Field” setting when the ToAddress was set to “literal”. LAL-4382
known issues/considerations
The Lavastorm Update for Salesforce node does not currently handle null entries for fields to be updated. LAL-5538
New in LAL 6.1.39 / LAL 6.0.39
Overview
New in Version 6.X.39 of LAL:
The Lavastorm Metadata for Salesforce node
The Lavastorm Update for Salesforce node
The Quantile Regression node (within the Statistical and Predictive Analytic Node Pack)
The Statistical and Predictive Analytic Node Pack (Powered by TIBCO®) is Premium licensed.
Note – The Quantile Regression node is supported on the Windows platform, RedHat and SUSE Linux 64bit platform only.
Details of New Functionality
The Lavastorm Metadata for Salesforce node retrieves metadata in SALESFORCE.COM objects via the SALESFORCE.COM REST API.
The Lavastorm Update for Salesforce node inserts, updates, upserts or deletes records in SALESFORCE.COM objects via the SALESFORCE.COM REST API.
The Quantile Regression node models data using quantile regression allowing identification of data trends for conditional quantiles of a response variable distribution.
Bugfixes
The Excel File node default for the NoOutputPinErrorBehaviour parameter has been amended to log instead of Error LAL-5238 (LAL 6.1.39 only)
The Lavastorm Get for Salesforce node now supports queries for parent-to-child and child-to-parent relationships. LAL-5282
The Lavastorm Transact for Salesforce node requested a new session on each iteration, this resulted in BRE hanging if more than 10 instances were running. The new suite of Lavastorm for Salesforce nodes (Get, Metadata and Update) all output the session details so the same session can be used repeatedly. LAL-5077
The HTTP node now masks the TrustStoreFilePassword parameter LAL-5278
See the release notes for more details. the Release notes and download can be found on our download page.
LAL 6.1.41 (Compatible with LAE 6.1.x) and LAL 6.0.41 (Compatible with LAE 6.0.x)
Hi all,
New in LAL 6.1.41 / LAL 6.0.41
Overview
The Cleanse String node enhancement
The Data Converter node (Experimental)
The Lavastorm Get / Metadata / Update for Salesforce node enhancement
The Select Fields node enhancement
The Spark Download node (Experimental)
The Spark End Session node (Experimental)
The Spark Java Submit node (Experimental)
The Spark Sort node (Experimental)
The Spark Start Session node (Experimental)
The Spark Upload node (Experimental)
Details of New Functionality
The Cleanse String node has an improved performance. LAL-5643
The Data Converter node converts the data type of each field based on the discovered data type.
The Lavastorm Get / Metadata / Update for Salesforce nodes now support the use of a Proxy server. LAL-5556 / LAL-5557 / LAL-5558.
The Select Fields node has an improved performance. LAL-5644
The Spark Download node downloads data in the Avro file format from a Spark cluster.
The Spark End Session node ends a Spark session.
The Spark Java Submit node provides an interface with a Spark cluster to execute a Spark application.
The Spark Sort node sorts data in the Avro file format using a Spark cluster.
The Spark Start Session node starts a Spark session.
The Spark Upload node uploads data to a Spark cluster in the Avro file format.
Bugfixes
The Output Excel and Append Excel nodes now cater for dates greater than the year 2098. LAL-5563
The Change Metadata node now caters for dates greater than the year 2098. LAL-5507
Third party known issues/considerations
The Avro 1.7.7 specification calls out some constraints on the metadata that it can support. Specifically it places restrictions on the names of fields:
The Field names must:
start with [A-Za-z_]
subsequently contain only [A-Za-z0-9_]
Avro 1.7.7 does not support date, time and datetime data types. As a result if you wish to upload data and use the Lavastorm nodes then these fields will need to be converted to string.
See the release notes for more details. the Release notes and download can be found on our download page.
Example graphs attached for the Spark based nodes. Please read the Spark Configuration v0.1.pdf before attempting to run the nodes.
Regards
Simon
Last edited by simon.wilby; 12-04-2015 at 02:45 PM.
Reason: corrected installers link
LAL 6.1.42 (Compatible with LAE 6.1.x) and LAL 6.0.42 (Compatible with LAE 6.0.x)
Hi all,
New in LAL 6.1.42 / LAL 6.0.42
Overview
Tibco Enterprise Runtime for R (TERR) embedded R engine has been updated (6.1 only)
The Spark Filter node (Experimental)
The Spark Java Submit node enhancement
The Spark Sample node (Experimental)
Details of New Functionality
The Tibco Enterprise Runtime for R (TERR) embedded R engine has been updated from 3.2 to 4.0 on both the Windows 64bit and Linux 64bit platforms. LAL-5550 (6.1 only)
The Spark Filter node filters data using a Spark cluster.
The Spark Java Submit node has been enhanced to allow customers to specify configuration properties for the Java application execution. LAL-5791
The Spark Sample node samples data using a Spark cluster.
Bugfixes
The Lavastorm Update for Salesforce node now advises which record has failed to update and stops executing at that record. LAL-5537
The Lavastorm Update for Salesforce node now completes updates where values are null. LAL-5538
See the release notes for more details. the Release notes and download can be found on our download page.
Example graphs attached for the Spark based nodes. Please read the Spark Configuration v0.1.pdf before attempting to run the nodes.
LAL 6.1.43 (compatible with LAE 6.1.x) and LAL 6.0.43 (Compatible with LAE 6.0.x)
Hi all,
New in LAL 6.1.43 / LAL 6.0.43
Overview
The Lavastorm Get for Salesforce node enhancement
The Spark Agg node (Experimental)
The Spark Data Viewer node (Experimental)
The Spark Join node (Experimental)
Spark nodes enhancement
Details of New Functionality
The Lavastorm Get for Salesforce node now supports the SOQL convertCurrency and date functions. LAL-5871
The Spark Agg node aggregates data using a Spark cluster.
The Spark Data Viewer node previews data from Spark.
The Spark Join node joins data using a Spark cluster.
A number of Spark based nodes (i.e. those that include an InputSource parameter) have been enhanced to provide a default value for the InputSource parameter; this is so that users only need to connect the second output pin from a previous Spark node into the second input pin. LAL-5843
Bugfixes
The TIBCO Enterprise Runtime for R (TERR) based nodes (the Statistical and Predictive Analytic Node Pack) were failing to execute on the Windows 32bit operating system. The TERR remoteJvmArch property has been updated to specify the operating system bitness instead of the default 64bit option. LAL-5950 (LAL 6.1.43 only)
See the release notes for more details. the Release notes and download can be found on our download page.
Example graphs attached for the Spark based nodes. Please read the Spark Configuration v0.1.pdf before attempting to run the nodes.
LAL 6.1.45 (Compatible with LAE 6.1.x) and LAL 6.0.45 (Compatible with LAE 6.0.x)
Hi all,
New in LAL 6.1.45 / LAL 6.0.45
Overview
Please Note: All of the Spark nodes now require Spark 1.5.0 as a minimum in the <LAE Install>/lib/java/lal/spark directory. In addition the Spark SQL Query node requires the Hive Client Configuration to be located on the laeServer machine. (See the Spark Configuration Guide for Lavastorm.pdf included in your installed documentation).
The Archive Compress node
The QVD File node
The Spark SQL Query node (Experimental)
The Spark Agg node enhancement
The Spark Join node enhancement
The Excel File node enhancements
Details of New Functionality
The Archive Compress node creates a compressed archive file containing one or more files. LAL-4961
The QVD File node imports data from a QVD file. LAL-3102. Additional software libraries are required in order for this node to work (commons-io-2.4.jar, irbi-fileutils-0.1.0.jar, jdbm-3.0-SNAPSHOT.jar, qvd-converter-core-1.5.0.jar and qvd-reader-1.5.0.jar).
Please contact Ralf Becher using the contact details below in order to purchase these libraries. Once you have those libraries they need to be saved to <LAE install Directory>/lib/java/lal.
IT Beratung Ralf Becher
Leipzig, Germany ralf.becher@web.de
The Spark SQL Query node provides an interface with a Spark cluster to execute a Spark SQL query. LAL-6194
The Spark Agg node has new parameters included to allow customers to include or exclude fields from the aggregate function. LAL-5817
The Spark Join node has been enhanced to include new parameters so that users are able to filter on one or both datasets. LAL-6001
The Excel File node has a new parameter called IgnoreCellFormatting which allows the raw data underneath cell formatting to be imported to LAE. The default behavior is set to “Ignore” to allow for backwards compatibility. LAL-1554/gnats 5119
Bugfixes
The Trim Fields node now fully caters for fields with or without single quotes. LAL-5004
The Trim Fields node set to NULL if empty option has been improved to work on all fields. Also if the field names do not exist for the excluded/included option an exception is now returned. LAL-4049
Third party known issues/considerations
The Spark SQL Query node has highlighted some Apache issues with the following scenarios:
Selecting a field with binary type fails with the exception “UnresolvedUnionException: Not in union ["bytes","null"]”. This is already raised on Apache JIRA: https://issues.apache.org/jira/browse/AVRO-1401
Describe operation returns 3 fields (col_name, col_type, comment) however the comment field is handled as not “nullable” but returns a NULL value.
The Spark SQL Query node processes against Hive tables. When Hive tables are processed by the cluster, the minimum memory requirement is higher compared with the memory required to run other Spark nodes that do not access Hive. The DriverMemory and ExecutorMemory both have a minimum 5G threshold. Users are advised to increase this for larger environments.
New in LAL 6.1.44 / LAL 6.0.44
Overview
Please Note: All of the Spark nodes now require Spark 1.5.0 as a minimum in the <LAE Install>/lib/java/lal/spark directory (See the Spark Configuration Guide included in your installed documentation).
The Spark Sample node has been updated to return ‘N Records’ instead of ‘First N Records’. When using the 'N records' option, the order of the generated sample set is not guaranteed to reflect the order of the data in the input file(s).
Attached are example graphs for the Archive Compress and Spark SQL nodes
Spark Download and Spark Data Viewer string data type enhancement
Spark nodes delimited file support enhancements
Spark nodes GA
Details of New Functionality
The status of the (previously Experimental) Data Converter node has now been changed from experimental to fully supported.
The Spark Agg node now outputs a GroupBy field when the field is also included in an aggregation function. LAL-6480
The Spark nodes have been enhanced to offer an additional Kerberos authentication method which uses a secure keytab file.
The Spark Download node and Spark Data Viewer node have been enhanced to enable the customer to specify whether string fields should be output as unicode field type (default) or as string field type. LAL-6870
A number of Spark nodes have been enhanced to allow customers to specify a delimited file as the node’s input data source instead of the default Avro file format. Configuration options enable the customer to specify: the delimiter character, whether the delimited file has a header record, the quote character, the escape character and whether empty strings are to be considered NULL values. By default, the nodes attempt to automatically identify the schema. The schema can also be explicitly defined. LAL-6805, LAL-6697
The status of the (previously Experimental) Spark nodes has now been changed from experimental to fully supported. This relates to the following nodes: Spark Start Session, Spark End Session, Spark Upload, Spark Download, Spark Data Viewer, Spark Agg, Spark Filter, Spark Join, Spark Sample, Spark Sort, Spark SQL Query.
Bugfixes
Some extraneous parameters are no longer visible on the JSON Data node and the Logistic Regression Diagnostics node. LAL-6523, LAL-6840
Duplicate help text was removed from the Spark SQL Query node’s help information. LAL-6797
Removed Functionality
The following experimental nodes have been removed from the installation:
Complex Lookup
Date Range Lookup
String Based Lookup
Third party known issues/considerations
In the situation where a Spark node is configured to import data from a delimited file and the schema has been defined by the user, the schema must be valid. When the schema is invalid the node execution may complete successfully but no records will be output.
The Release Notes and installers can be found on our download page.
X-Ref MetaData node added - takes 2 inputs and performs a cross-reference against the metadata of the inputs, to find the common fields and types of the two inputs.
Last edited by gmullin; 02-17-2018 at 04:43 PM.
Reason: Removed external URL