EDL 3.3 Best Practices #5-8
This is a continued list of Best Practices for EDL 3.3.
#5 - PERFORMANCE SETTINGS - RECORD GOVERNOR
Often a customer will purchase EDL after having run QC-CALC for several years. At that point, there are potentially a lot of records to be loaded (sometimes 100,000's). To help control the data flow, a governor was added to only grab x records per round. By default this number is 1000 (see picture above), but this can be increased depending on your needs. In addition, the "Continue loading until files are complete" checkbox can help get the data in quicker than without it checked. For example, let's say a qcc file has 50,000 records in it and a customer sets up an EDL load event to run once per day at 2am. Based on the query governor at 1000 records, it will take 50 days to get all of the data into the database since it will only pull 1000 records per run. If the "Continue" checkbox is checked, it will continue grabbing the next 1000 records until the file is completely caught up.
Main Benefits: Gets it done quicker.
Main Cost: Not really any although smaller QCC files may get stuck behind larger files if they have to load many chunks of 1000 records.
#6 - EXPRESS LOADING (ONLY IF NEEDED)
Turning on this option (see picture above) will make loading larger lists of files faster. Normally, EDL checks each QCC file for new records since the last load. It then checks for changes to the EDL description, changes to the dimension fields, and edits to the records of the file. If express loading is turned on, it only checks if there are new records. If there are new records then the rest of the changes are loaded to the database at that moment. If there are no new records, none of the other changes are checked. The theory is if a QCC file is not currently active (no new records are arriving) then it's likely that it's not receiving updates either and should be skipped unless new records arrive.
Main Benefits: It can process larger lists of QCC files a lot faster than normal.
Main Cost: If a file is not receiving new records, updates that occur to the file are not loaded to the database until much later than when they occurred.
#7 - QCC FILE MONITORING
Internally QCC files are uniquely identified by a combination of their name and an internal creation date that QC-CALC inserts into the file. Whenever a QCC file is compacted or purged, QC-CALC resets the internal date. When EDL attempts to load a QCC file to the database, it looks up the file name and creation date. If found, it loads the data under the existing file. If not found, it creates a new QCC file.
Check for compacted/purged QCC files during load: If checked, when each QCC file is loaded, the normal check is performed. Whenever a QCC file with the same name and EDL description is found with a different date, it is assumed the user compacted or purged the file in QC-CALC and that this is really just a copy of the existing file. As such, in order to prevent a lot of duplication in the file lists of ERS, EDL can automatically set the archive flag of the existing QCC file and then upload the new file. EDL can also optionally email an administrator.
Check for renamed QCC files during load: If checked, when each QCC file is loaded, the normal check is performed. Whenever a QCC file has the same exact creation date as an existing QCC file within the same EDL description, it is assumed the file was simply renamed outside of QC-CALC (in Windows Explorer). When found, the existing file in the database is set to archived and the new file is loaded. EDL can also optionally email an administrator. It is considered extremely rare that two QCC files would be created at the same moment in time so the date comparison is considered to be safe.
The checkboxes pictured below are the recommended settings for QCC File Monitoring.
Main Benefits: The lists in ERS will be more manageable and less cluttered when duplicate files are flagged as archived.
Main Costs: None. If there are any issues where EDL archives the wrong file, they can easily be fixed in the ERS Administrator.
#8 - EMAIL NOTIFICATION
This is one of the most important features to set up in EDL. Because EDL is a service that is likely running on a PC far from the user, the only way it can communicate what is going on is to reach out via email. If email notifications are not set up, EDL will continue running but will be unable to notify admins when exceptions occur.
First set up the SMTP account that will be used to send the emails. This information can be obtained from your local IT department. Just ask them for the SMTP server name, the port number to use and a user account and password that is acceptable. If user name and password is not required, simply enter "test" in both the username and password boxes. Click OK when finished so the changes are committed. Do this before updating the notifications.
Next go to the Notifications screen and enter an email address that will receive error messages when they occur. You can also optionally add an email address to the field at the bottom so EDL emails when it is successful. Use this field only if you are running the event on a longer period, such as one per day or enjoy a full inbox.
Main Benefit: Notification when an exception occurs.
Main Costs: Extra emails if you have a lot of exceptions.