3.9 Expiry
In addition to changing the precision of data stored in the system, another option for limiting the amount of data is changing the expiry times.
Expiry times establish database retention times. Longer expiry times mean the data will be kept in the system longer. Setting longer expiry times leads to larger local datastores and longer data processing times. Expiry times are defined as followed:
- Unit: Second, Minute, Hour, Day, Week
- Multiplier: Defines the number of units given above
- Divider: Defines fraction of units
A process, called the MarkedRecordManager, labels the data that needs to be removed by the Rolling Barrel. The Rolling Barrel task removes the data once it reaches its expiry time.
Set expiry times for workflows by editing WorkflowDescriptors.xml in the RegionConfigFiles directory.
Rolling Barrel:
Recall from Introduction to CHPS that the Rolling Barrel task makes a new data store copy and then deletes the old one. It then moves the newly created data store to its place. This process prevents the system from becoming overpopulated.
DbVis
Use DbVis to search by expiry times and see what is marked for deletion at a certain time.
Open the DbVis interface and use the SQL query: Select <table> WHERE expirytime=’YYYY-MM-DD HH:mm:SS’
See the youtube video on the right. -------->
Options for Optimizing
The Configuration Focal Point can decide which information can be assigned shorter expiry times. The default time is set at 30 days. Many time series, however, are only needed for a few model runs. Work with your RFC to identify data stored in the database too long.
Example: ESP (Extended Streamflow Prediction) runs generate a large quantity of data, most of which is only needed for the forecast runs.
Try marking the ESP data for deletion soon after the forecast period.
Schedule ESP runs early in the morning before the forecast period, keep the data for 5 or 6 hours, and mark the expiry time for shortly before the Rolling Barrel task.
The Rolling Barrel then marks the data for deletion, and it does not needlessly synch.
For details on setting expiry times, use the job sheets below.