Task Manager Configuration

You can use the configuration parameters in create_config.py to configure the MATRIXX Task Manager, which scans the databases on a scheduled interval for tasks that require processing by MATRIXX Charging Application.

Each Task Manager on a processing pod initiates processing of several subscriber management-related tasks, including:
  • Deleting old sessions and old event objects Event Detail Record (EDRs).
  • Sending notifications.
  • Canceling product offers.
  • Removing expired purchased offers.
  • Removing expired balances.
  • Performing recurring processing.
  • Handling multi-transaction processes, such as Pay Now authorization objects.
  • Handling some system upgrade operations.
To start the work, the Task Managers scan the local database set on a configured interval. When objects are found that require processing, the work messages are sent to MATRIXX Engine, which performs the cleanup and business operations.

Because the Task Manager does all task processing and sends messages to its local MATRIXX Charging Application, it is possible for the engine to become overloaded, for example, if a spike of activity is scheduled to occur. To handle such cases, there are sets of parameters to configure how to throttle the processing rate by message type, per blade, and per engine.

Typically, only the server throttling parameters are used; however, there are instances in which both parameters in a set are used. This might occur during spikes in activity or when a processing server is added to the system topology. When configuring the per-server and per-engine values, take into consideration possible changes in engine topology.

Dynamic throttling of internal load can also be enabled, based on monitored task manager resource usage.

Task Manager Configuration Parameters lists the information required to configure the Task Manager. The MATRIXX Charging Application server is the Charging Server.
Note: In bare metal environments, MATRIXX components and scripts, such as the processing server or print_blade_stats.py, run on a server, sometimes called a blade. To avoid confusion, some scripts and code samples that refer to blades have not changed in cloud native environments. When possible, MATRIXX documentation uses cloud native terminology, for example, when referring to processing pods.
Table 1. Task Manager Configuration Parameters
Parameter Description Default Value
Event Generation Trigger Delay The amount of time to wait, in seconds, at the end of a period before triggering event generation.

create_config.info question: How long (in seconds) do you want to delay at the end of a period before triggering event generation?

600
Audit Data Creation Answer y to this question to write additional usage data to the transaction record when an aggregation object is updated or when a usage event is created.

create_config.info question: Should audit data be generated for all usage transactions (y/n)?

n
Notification Resend Attempt Duration The maximum time, in minutes, to continue to resend missed notifications when acknowledgment of delivery is not received.

create_config.info question: What is the maximum time in minutes do you want to allow to send missed notifications?

 14400
Subscriber Database — Per-Server Task Message Maximum The maximum number of subscriber database task messages, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many subscriber database task messages should be sent per second per blade?

 250
Subscriber Database — Engine Task Message Maximum The maximum number of subscriber database task messages to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many subscriber database task messages should be sent per second in the engine?

 750
Subscriber Database — Task Message Latency Maximum The maximum amount of time in milliseconds between subscriber database task messages before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for subscriber database task messages before logging an error?

 18000000
Subscriber Database — Task Message Window Size This is the time slice to use to determine if the maximum subscriber database task message rate is exceeded. This is valid only if the maximum message rate in messages per second is nonzero. The minimum value is 100 and the maximum value is 2000000 (2 seconds).

The maximum message rate is checked by calculating the maximum number of messages that can be received during the specified time slice. After this calculated value is met, all remaining messages received during this time slice are rejected. It is important to set this value correctly. Setting a window size that is too small might cause a small spike of messages to be rejected, even though the total number in a second has not been exceeded.

Setting a window size that is too large might cause a lot of dead time where all messages are rejected until the next time slice. This happens when spikes early in the time slice cause the calculated maximum to be reached.

create_config.info question: TaskMgr:What is the window size (in microseconds) to use when calculating the subscriber database message rate?

 1000000
Subscriber Database — Per-Second Database Scan The amount of time to wait, in seconds, before rescanning the subscriber database for tasks that require processing.

create_config.info question: TaskMgr:How often (in seconds) should the subscriber database scanner be run?

 30
MtxTaskObjects — Per-Server Message Maximum The maximum number of messages for MtxTaskObjects, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many messages for MtxTaskObjects should be sent per second per blade?

 100
MtxTaskObjects — Per-Engine Message Maximum The maximum number of messages for MtxTaskObjects to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many messages for MtxTaskObjects should be sent per second in the engine?

 200
MtxTaskObjects — Message Latency Maximum The maximum amount of time in milliseconds between messages for MtxTaskObjects before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for messages for MtxTaskObjects before logging an error?

 250
MtxTaskObjects — Post-Scan Pause Duration The amount of time to wait, in seconds, before rescanning the database for MtxTaskObjects that require processing.

create_config.info question: TaskMgr:How long (in microseconds) should the schedule_db_task_object_scan_task pause after completing a scan?

 10000
MtxTaskObjects — Resend Delay The amount of time to wait, in microseconds, before re-sending messages for MtxTaskObjects for processing.

create_config.info question: TaskMgr:How much time should elapse before resending messages for MtxTaskObjects (in microseconds)?

 5000000
MtxApplicationDelayResponseObjects — Per-Server Message Maximum The maximum number of messages for MtxApplicationDelayResponseObjects, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many messages for MtxApplicationDelayResponseObjects should be sent per second per blade?

 100
MtxApplicationDelayResponseObjects — Per-Engine Message Maximum The maximum number of messages for MtxApplicationDelayResponseObjects to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many messages for MtxApplicationDelayResponseObjects should be sent per second in the engine?

 200
MtxApplicationDelayResponseObjects — Message Latency Maximum The maximum amount of time in milliseconds between messages for MtxApplicationDelayResponseObjects before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for messages for MtxApplicationDelayResponseObjects before logging an error?

 250
MtxApplicationDelayResponseObjects — Post-Scan Pause Duration The amount of time to wait, in microseconds, before rescanning the database for MtxApplicationDeleayResponseObjects that require processing.

create_config.info question: TaskMgr:How long (in microseconds) should the schedule_db_application_delay_response_object_scan_task pause after completing a scan?

 10000
MtxApplicationDelayResponseObjects — Resend Delay The amount of time to wait, in microseconds, before resending messages for MtxApplicationDelayResponseObjects for processing.

create_config.info question: TaskMgr:How much time should elapse before resending messages for MtxApplicationDelayResponseObjects (in microseconds)?

 5000000
MtxCallOutObjects — Per-Server Message Maximum The maximum number of messages for MtxCallOutObjects, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many messages for MtxCallOutObjects should be sent per second per blade?

 0
MtxCallOutObjects — Per-Engine Message Maximum The maximum number of messages for MtxCallOutObjects to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many messages for MtxCallOutObjects should be sent per second in the engine?

 0
MtxCallOutObjects — Message Latency Maximum The maximum amount of time in milliseconds between messages for MtxCallOutObjects before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for messages for MtxCallOutObjects before logging an error?

 250
MtxCallOutObjects — Post-Scan Pause Duration The amount of time to wait, in microseconds, before rescanning the database for MtxCallOutObjects that require processing.

create_config.info question: TaskMgr:How long (in microseconds) should the schedule_db_call_out_object_scan_task pause after completing a scan?

 10000
MtxCallOutObjects — Resend Delay The amount of time to wait, in microseconds, before resending messages for MtxCallOutObjects that require processing.

create_config.info question: TaskMgr:How much time should elapse before resending messages for MtxCallOutObjects (in microseconds)?

 1000000
MtxPaymentOpObjects — Per-Server Message Maximum The maximum number of messages MtxPaymentOpObjects, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many messages for MtxPaymentOpObjects should be sent per second per blade?

 100
MtxPaymentOpObjects — Per-Engine Message Maximum The maximum number of messages MtxPaymentOpObjects to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many messages for MtxPaymentOpObjects should be sent per second in the engine?

 200
MtxPaymentOpObjects — Message Latency Maximum The maximum amount of time in milliseconds between messages for MtxPaymentOpObjects before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for messages for MtxPaymentOpObjects before logging an error?

 250
MtxPaymentOpObjects — Post-Scan Pause Duration The amount of time to wait, in microseconds, before rescanning the database for MtxPaymentOpObjects that require processing.

create_config.info question: TaskMgr:How long (in microseconds) should the schedule_db_payment_op_object_scan_task pause after completing a scan?

 10000
MtxPaymentOpObjects — Resend Delay The amount of time to wait, in microseconds, before resending messages for MtxPaymentOpObjects for processing.

create_config.info question: TaskMgr:How much time should elapse before resending messages for MtxPaymentOpObjects (in microseconds)?

 5000000
MtxNotificationObjects — Per-Server Message Maximum The maximum number of messages for MtxNotificationObjects, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many messages for MtxNotificationObjects should be sent per second per blade?

 500
MtxNotificationObjects — Per-Engine Message Maximum The maximum number of messages for MtxNotificationObjects to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many messages for MtxNotificationObjects should be sent per second in the engine?

 1000
MtxNotificationObjects — Message Latency Maximum The maximum amount of time in milliseconds between messages for MtxNotificationObjects before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for messages for MtxNotificationObjects before logging an error?

 6000
MtxNotificationObjects — Post-Scan Pause Duration The amount of time to wait, in microseconds, before rescanning the database for MtxNotificationObjects that require processing.

create_config.info question: TaskMgr:How long (in microseconds) should the schedule_db_notification_object_scan_task pause after completing a scan?

 10000
MtxNotificationObjects — Resend Delay The amount of time to wait, in microseconds, before resending messages for MtxNotificationObjects.

create_config.info question: TaskMgr:How much time should elapse before resending messages for MtxNotificationObjects (in microseconds)?

 5000000
MtxStartCallObjects — Per-Server Message Maximum The maximum number of messages for MtxStartCallObjects, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many messages for MtxStartCallObjects should be sent per second per blade?

 0
MtxStartCallObjects — Per-Engine Message Maximum The maximum number of messages for MtxStartCallObjects to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many messages for MtxStartCallObjects should be sent per second in the engine?

 0
MtxStartCallObjects — Message Latency Maximum The maximum amount of time in milliseconds between messages for MtxStartCallObjects before generating an error.

create_config.info question: TaskMgr:What is the maximum latency (in milliseconds) for messages for MtxStartCallObjects before logging an error?

 250
MtxStartCallObjects — Post-Scan Pause Duration The amount of time to wait, in microseconds, before rescanning the database for MtxStartCallObjects that require processing.

create_config.info question: TaskMgr:How long (in microseconds) should the schedule_db_start_call_object_scan_task pause after completing a scan?

 10000
MtxStartCallObjects — Resend Delay The amount of time to wait, in microseconds, before resending messages for MtxStartCallObjects that require processing.

create_config.info question: TaskMgr:How much time should elapse before resending messages for MtxStartCallObjects (in microseconds)?

 1000000
Activity Database Task — Per-Server Message Maximum The maximum number of activity database task messages, per blade, to send per second for processing.

create_config.info question: TaskMgr:How many activity database task messages should be sent per second per blade?

 250
Activity Database Task — Per-Engine Message Maximum The maximum number of activity database task messages to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many activity database task messages should be sent per second in the engine?

 500
Activity Database Scan Interval The amount of time to wait, in seconds, before rescanning the activity database for operations that require processing.

create_config.info question: TaskMgr:At what interval (in seconds) should the activity database be scanned for operations that require processing?

 60
Activity Database Task — Per-Server Pre-Pause Message Maximum The number of outstanding activity database task messages, per server, to send before pausing.

create_config.info question: TaskMgr:How many outstanding activity database task messages should be sent per blade before pausing?

 64
Activity Database Task — Per-Engine Pre-Pause Message Maximum The number of outstanding activity database task messages to send for all the servers in the engine before pausing.

create_config.info question: TaskMgr:How many outstanding activity database task messages should be sent in the engine before pausing?

 128
Event Database Task — Per-Server Message Maximum The maximum number of event database task messages, per server, to send per second for processing.

create_config.info question: TaskMgr:How many event database task messages should be sent per second per blade?

 2500
Event Database Task — Per-Engine Message Maximum The maximum number of event database task messages to send per second for processing, for all the servers in the engine.

create_config.info question: TaskMgr:How many event database task messages should be sent per second in the engine?

 5000
Event Database Task — Message Window Size This is the time slice to use to determine if the maximum event database task message rate is exceeded. This is valid only if the maximum message rate in messages per second is nonzero. The minimum value is 100 and the maximum value is 2000000 (2 seconds).

The maximum message rate is checked by calculating the maximum number of messages that can be received during the specified time slice. After this calculated value is met, all remaining messages received during this time slice are rejected. It is important to set this value correctly. Setting a window size that is too small might cause a small spike of messages to be rejected, even though the total number in a second has not been exceeded.

Setting a window size that is too large might cause a lot of dead time where all messages are rejected until the next time slice. This happens when spikes early in the time slice cause the calculated maximum to be reached.

create_config.info question: TaskMgr:What is the window size (in microseconds) to use when calculating the event database task message rate?

 1000000
Event Database — Database Scan Pause Duration The amount of time to wait, in microseconds, before rescanning the event database.

create_config.info question: TaskMgr:How long (in microseconds) should the event database scanner pause between each scan?

 5000
Event Database Task — Per-Server Pre-Pause Message Maximum The number of outstanding event database task messages, per server, to send before pausing.

create_config.info question: TaskMgr:How many outstanding event database task messages should be sent per blade before pausing?

 48
Event Database Task — Per-Engine Pre-Pause Message Maximum The number of outstanding event database task messages to send before pausing, for all the servers in the engine.

create_config.info question: TaskMgr:How many outstanding event database task messages should be sent in the engine before pausing?

 128
Single-Transaction Event Delete Maximum The maximum number of events that can be deleted in a single transaction.

A limitation exists on the number of events that can be deleted in a single transaction. Making this number large reduces the message traffic between the Task Manager and the Charging Server, but it also increases the chance of business conflicts and can make the footprint of the message size at transport latent.

create_config.info question: TaskMgr:What is the maximum number of events that can be deleted in a single transaction?

 100
Event Database — Pre-Deletion Maximum Database Size Configures the event cleanup scanner to run when the event database reaches this size, in bytes, to prevent it from growing out of space. When the event cleanup scanner runs, it deletes event objects in order of their creation, oldest to newest.

create_config.info question: TaskMgr:How large (in bytes) should the event database grow before events are deleted?

This value must be smaller than the initial size of the event database, which is determined by multiplying two parameters in the Database Sizing Configuration section: Number of Event Objects and Event Object Average Size.

MATRIXX suggests setting the threshold 10–20% smaller than the initial event database size. For example, to keep 3.3 million events in the database with an average event size of 1200 bytes (3960000000 bytes), the threshold must be around 3600000000. If it is configured to be larger than the initial event database size, it must be smaller than the difference between the maximum event database size and 1 extended data segment size.

During system configuration, the create_config.py script verifies that this cleanup threshold value is less than the difference. If it is more than the allowed value, the create_config.py exits with an error similar to the following error:
Error: based on event database configuration
(maxSize=65127055360 bytes), event cleanup purge threshold 
cannot be more than 65063454720 bytes

In this case, change the answer to this question to a value that is less than or equal to the value indicated in the error message.

 117964800
Upgrade Cleanup — Per-Server Message Maximum The maximum number of upgrade cleanup requests that are in-flight to the Charging Server at one time, from a server. The number of outstanding cleanup requests to the Charging Server can throttle the event cleanup operation on the server. When this number is reached, the Task Manager pauses until responses have been received from the Charging Server acknowledging that the work is complete.

create_config.info question: TaskMgr:How many upgrade cleanup messages should be sent per second per blade?

 1000
Upgrade Cleanup — Per-Engine Message Maximum The maximum number of upgrade cleanup requests that are in-flight to the Charging Server at one time, from all the servers in the engine. When this number is reached in the engine, the Task Manager pauses until responses have been received from the Charging Server acknowledging that the work is complete.

create_config.info question: TaskMgr:How many upgrade cleanup messages should be sent per second in the engine?

 2000
Upgrade Cleanup — Message Window Size This is the time slice to use to determine if the maximum upgrade cleanup task message rate is exceeded. This is valid only if the maximum message rate in messages per second is nonzero. The minimum value is 100 and the maximum value is 2000000 (2 seconds). The maximum message rate is checked by calculating the maximum number of messages that can be received during the specified time slice. After this calculated value is met, all remaining messages received during this time slice are rejected. It is important to set this value correctly. Setting a window size that is too small might cause a small spike of messages to be rejected, even though the total number in a second has not been exceeded. Setting a window size that is too large might cause a lot of dead time where all messages are rejected until the next time slice. This happens when spikes early in the time slice cause the calculated maximum to be reached.

create_config.info question: TaskMgr:What is the window size (in microseconds) to use when calculating the upgrade cleanup task message rate?

 100000
Dynamic Throttling Answer y to this question to enable dynamic throttling for Task Manager based on resource usage from System Monitor. This feature monitors traffic for a specified server, and, depending on values specified in the mtx_config_base.xml file, throttles internal traffic generated by Task Manager as needed.

For more information about dynamic throttling parameters, see the discussion about Task Manager dynamic load throttling configuration.

create_config.info question: TaskMgr:Do you want to enable dynamic throttling feature (y/n)?

y
Maximum Active Sessions The maximum number of active sessions to keep running.

create_config.info question: What is the maximum number of active sessions?

 100000
Database Garbage Collection Settings Answer y to keep the default database garbage collection settings.

create_config.info question: Do you want to use the default database garbage collection settings?

 y
Shared Memory Size Specify the shared memory size in MB to use.

create_config.info question: What is the shared memory size in MB to use?