Resolved Issues
MATRIXX version 5290 includes the following resolved issues.
Resolved issues include the following:
Platform
| Issue Number | Description |
|---|---|
| MTX-70242 | Pricing loader failed in Engine Operator-based deployments because it was unable to access the Kubernetes API. |
| MTX-70228 | Pricing loader failed in Engine Operator-based deployments because the MTX_VERSION environment variable was not set. |
| MTX-69750 | A event unique sequence ID might not be assigned in order of LocalSequence in a work order. |
| MTX-69602 | The previous default amount of reserved non-JVM memory, 256mi, was likely to be inadequate in most scenarios. It has been increased to 1024Mi. |
| MTX-69409 | Cluster Manager did not report anything in logs when a connection to a peer is restored after failure. |
| MTX-69303 | Due to a rare timing issue, if parallel_init_db was disabled and the standby publishing server on the standby engine joined the topology in the
PRE_INIT state, it sometimes caused the publishing cluster to be unable to transition to the next state. |
| MTX-69285 | There was an issue that resulted in a missing file extension being created for price plan download ZIP files and missing price plan reference data from those ZIP files. |
| MTX-69271 | The engine.additionalContainers Helm configuration did not work if using engine operator with mtx-engine-mgr as the engine manager. |
| MTX-69170 | In an engine-operator deployment, when a process was killed in the main container of an engine or Traffic Routing Agent (TRA-PROC or TRA-PUB) pod causing the container to restart, the engine
did not auto heal as expected if engine.additionalContainers was configured and the name of at least one of the additional containers was
alphabetically before the name of the main container of this pod. |
| MTX-69166 | Due to a rare timing issue, replay to the standby engine might stop after an engine switchover. |
| MTX-69111 | If the external CAMEL Gateway was running in a separate pod, cluster manager on the engine pod sometimes gave an error when trying to enqueue a message to CAMEL Gateway that was not running inside the engine pod. |
| MTX-69001 | The Transaction Server assumed customers upgrading from release 5273 used push model so it required one more engine restart to use the pull model. |
| MTX-68827 | In an extreme edge case, when the Transaction Server was processing a large work order with a specific combination of MDC and balance set actions, a non-leader node did not handle the boundary condition correctly. This caused a server/pod failure of the non-leader node. |
| MTX-68757 | There was an issue related to configuring the service annotations for the cgw-m3ua services
(global.topology.domains[x].subdomains[y].engines[z].camelGateway.serviceAnnotations). |
| MTX-68741 | There was an issue related to configuring the service annotations for the proc-m3ua services
(global.topology.domains[x].subdomains[y].engines[z].processing.serviceAnnotations and
global.topology.domains[x].subdomains[y].engines[z].processing.m3ua.serviceAnnotations.) |
| MTX-68469 | Engine Health Checker logs contained more information than necessary. It was not possible to set a specific logging level. The engine.operatorV2.engineHealthChecker.containers.clusterMonitor.logLevel property has been added to specify the level of logging information. |
| MTX-68460 | The position of the related message ID was not consistent across all log messages. |
| MTX-68459 | The processing pod failed if the publishing cluster or standby engine failed in a PRE_INIT state before opening a transaction stream to the processing cluster. |
| MTX-68393 | Occasionally, the transaction replay pull model did not send the replay batch to the destination cluster, which resulted in performance degradation. |
| MTX-68309 | Prometheus did not show CAMEL Gateway pods. |
| MTX-68277 | When upgrading from prior releases, the push model was used by default. Now the standby engine can ask the source engine to use either the push or pull model for greater flexibility. |
| MTX-68258 | Gateway Proxy image set default system properties for engine host, port, timeout, and schema versions. |
| MTX-68085 | The standby engine did not send missing GTC requests correctly if using the replay push model, which kept the standby engine from catching up after that. |
| MTX-68050 | Due to a rare timing issue, when the standby engine was in transition to ACTIVE state, processing pods could complete replay syncing at slightly different times which could cause a few replay transactions to be rejected. |
| MTX-67873 | The event_archiver_task tried repeatedly to recover the same GTC. |
| MTX-67730 | There was a schema updating issue that resulted in RS Gateway losing information about fields that were marked for deletion in newer releases and losing information about new fields in data containers after synchronizing with older engines that did not have those fields. These errors happened post-upgrade when RS Gateway was communicating with the newly upgraded engine and it responded with data container parsing errors. |
| MTX-67658 | Sometimes, if the Transaction Server non-leader node could not commit a transaction, it resulted in pod failure. |
| MTX-67568 | In a cloud environment, if the engine was started in standby and failed in the middle of database initialization, the engine could start with an empty or incomplete database. check_engine_start_prereqs.py now detects this condition and prevents the engine from starting up with an empty or incomplete database. |
| MTX-67485 | The MEF publish report was getting deleted. |
| MTX-67258 | In a cloud environment, Route Cache Prometheus metrics were missing from the TRA-RT pod. |
| MTX-66888 | The network enabler pod was failing to start in Kubernetes, if multiple sub-domains were configured. |
| MTX-66514 | When a standby engine started up, occasionally the max object ID was not set on the checkpointing pod if the database was empty and without any load. This might have resulted in using more resources on the checkpointing pod when building up the OID index map after load began. |
| MTX-65572 | Transactions executed on a non-leader node gave inconsistent object views that sometimes caused the Charging Server to not detect the inconsistency and retry a request. |
| MTX-64713 | The MEF daily report feature depended on MEF files to be available on the publishing host for processing. |
| MTX-64696 | The analyze_transaction_logs.py script was slow. Now, it is much faster when analyzing many transaction logs. |
| MTX-64492 | After disabling event object creation, event object creation could not be re-enabled without engine cold restart. |
| MTX-63076 | The monitoring label configuration in Helm (global.monitoring.labels) was incorrectly applied to the engine, Traffic Routing Agent (TRA) load
balancer, TRA-DR, and TRA-RT service monitors. |
Pricing, Policy, and Rating
| Issue Number | Description |
|---|---|
| MTX-69842 | The Copy button was incorrectly disabled when copying a balance. |
| MTX-69569 | The Balance Distribution normalizer included an unsupported global option. |
| MTX-69562 | When ActivateAllOffer status action was triggered from the subman request PurchaseOffer or ModifyStatus, if ActivateAllOffer failed, the offer being purchased was committed anyway. |
| MTX-69508 | The end time for a purchased item was extended with an EndType of cycle_count even when the purchased item was inactive. |
| MTX-69502 | Trace messages generated by data modify actions did not correlate dynamically created pricing object ids to the source pricing object ids they were generated from. |
| MTX-69466 | An attempt to modify the tenantId using MtxRequestSubscriberModify or similar requests was rejected if the tenantId provided was the same tenantId already set in the object. |
| MTX-69115 | Commercial users could view the differences in objects on the synchronize changes page. This allowed one tenant to view changes made by another tenant. |
| MTX-69088 | Tier values were not set on balance difference normalizers when creating a pricing file. |
| MTX-69067 | When a suspended purchased item transitioned to inactive state by MtxStatusConditionPeriodExpiration, the IsSysInit flag was incorrectly set in MtxPurchasedItemStatusChangeEvent. |
| MTX-69011 | If a balance was an on-demand balance, the Cycle Start Type options were not disabled. |
| MTX-68998 | The create_config.py script did not generate mtx_config.xml correctly for selective update parser configuration. |
| MTX-68957 | MtxOfferTransitionToRecoverableNotification had an incorrect NotificationType value. |
| MTX-68690 | ESF3 and Rerating Engine pods now mount necessary files using PV shared storage. Previously, hostPath was used on the host machine. |
| MTX-68673 | The scaleBasis property of type Matrix causes an exception at start-up if set to null. |
| MTX-68610 | A balance was not selected for charges if it was at its credit limit even if the charge was entirely discounted. Now, such a balance can be chosen for the charge if the charge is completely discounted. |
| MTX-68570 | A usage rating with parallel beats would stop applying discounts for the remainder of a larger secondary beat after completing a smaller primary beat. |
| MTX-68532 | There was an issue with balance/meter thresholds with associated grant price components. In certain cases, the grant was applied multiple times, before the balance or meter reached the threshold and again when reaching the threshold. |
| MTX-68207 | Large price plans that crossed the 2^31 - 1 (2gb) byte length could not be created. |
| MTX-68204 | A balance rollover event was generated when there was no prior balance period to rollover. |
| MTX-68134 | Cached normalization results were incorrectly used for normalizers that use parameters from the purchased item, thereby producing incorrect results if these normalizers were invoked when applied for multiple purchased items. |
| MTX-68114 | publish_mefs.py sometimes failed with the error bash: /usr/bin/rsync: Argument list too long when trying to rsync a subset of
many files from the target publish directory. |
| MTX-68065 | For pre-active offers, the start and end dates if set were not returned as part of an owner query. |
| MTX-68042 | PurchaseOffer and ResumeOffer did not generate a recurring failure event if recurring failure was allowed and recurring processing failed due to insufficient funds. PurchaseOffer did not populate the offer status fields in the purchase event after the grace period profile status transition had occurred. |
| MTX-68385 | MATRIXX Engine image required files were missing. The rerating_db_info.yaml, credentials.json or hbase-site.xml, and stream_event_processing.yaml files must be in /opt/mtx/conf for the checkpoint pod to ensure that the snapshot loader works correctly. |
| MTX-68269 | If Keycloak was enabled, it was possible to change roles in My MATRIXX. Now, changes to the user roles from in My MATRIXX are disabled. All changes to roles must be done in Keycloak. |
| MTX-68041 | In generated events that included carrier taxes, the GL info reference from the taxes was not set. |
| MTX-67872 | When a URL is submitted that contains a domain name, the My MATRIXX checks if that domain exists and the user has access to it. If not, access is forbidden. The exceptions to this are when a user is creating or importing a new domain. Previously, there was no allow list. Now, there is an "allowedDomainUrls" allow list in matrixx.yaml to specify domain URLs that should not have the access check performed. |
| MTX-67546 | Trying to access cache-deleted database objects during event generation for balance transfer request failures caused errors. |
| MTX-66898 | There was a race condition between recurring and maintenance such that if recurring advances the billing cycle before maintenance was triggered, then the wallet was
left in a bad state where OriginalCycleEndTime was present but LastBillingCycleDefn was not because the advance billing cycle would wipe out the LastBillingCycleDefn.
In this state, terminateBillingCyclePeriods() during maintenance would complain with LM_ERROR Cannot find last billing cycle definition. |
| MTX-66297 | The replay stats from print_blade_stats.py did not update correctly after the engine state changed. |
| MTX-65160 | An error was not printed when an MtxPricingMetadata exception occurred. |
| MTX-64603 | Rows returned when querying a matrix using the REST API had wrong value_name values. |
| MTX-64549 | When configured to keep Rx application sessions upon termination of their underlying Gx policy sessions, a retained Rx session could result in failure to process RAA messages of unrelated Gx sessions and lead to their termination as well. |
| MTX-63801 | If the Inbound Roaming Subscription Template contained a meter balance, it was incorrectly propagated to a new Inbound Roamer Subscription. |
| MTX-63421 | Expected errors were not reported when an offer with a DMA containing a Source Normalizer Value and a Destination Type Workspace is purchased. |
Application Integration
| Issue Number | Description |
|---|---|
| MTX-70320 | Event Streaming Framework did not resume processing after engine restart. |
| MTX-69494 | The system attempted to recover the same event multiple times. |
| MTX-68716 | The event-purge job failed to execute the event_repository_purge script on the publishing pod. |
| MTX-68105 | The context path of mtx_event_streamer was incorrectly determined when running in Kubernetes. Now, mtx_event_streamer is the
context path whether on bare metal or Kubernetes. This context path is used during property resolution and in setting properties with -D switches, so
-Dmtx_event_streamer should be the prefix for all deployments (cloud and bare metal). |
| MTX-64739 | MEF or event loading or event streaming could be stuck when trying to recover a missing GTC detected in SEF files. |
| MTX-57583 | Under rare condition, the event stream reader attempted to read the next event in an SEF file that was purged causing event streaming to stop. |
Network Integration
| Issue Number | Description |
|---|---|
| MTX-69909 | The base mapping structure for SBA mapping extensions was changed erroneously. |
| MTX-69428 | The com_matrixx_cdr_producer_pending metric was not included in the documentation. |
| MTX-69269 | The value of maxConcurrentStreams was increased to a much larger value. This could cause performance issues under high load. The default value has
been set back to the original value of 100. |
| MTX-69183 | The mtx_sba_engine_client_connection_counter metric did not report as many engine connections as there actually were. |
| MTX-69109 | Outstanding engine notifications were processed while the Charging Function (CHF) was still starting. |
| MTX-68618 | When the downstream certificate was invalid, a connection error message was not reported in SBA Gateway logs. |
| MTX-68484 | The default settings for CDR_FILE_LOG_MAX caused network CDR logs to be lost when there was an extended Kafka outage and write to disk was enabled. The default values have been increased to mitigate this issue. |
| MTX-68464 | A non thread-safe variable was used for the writeQueueMaxSize threshold, resulting in SBA Gateway constantly writing to disk. A new com.matrixx.cdr.producer.pending
metric has been added to expose the value of the pending message variable. |
| MTX-68399 | The NRF Client was unable to target any NF services pods in the namespace through configurable label selectors. |
| MTX-68135 | The CDR Aggregation Function did not calculate the correct duration for an aggregated CDR. |
| MTX-68058 | A request sent to the engine that could not be processed, resulting in the engine not responding to the request, caused a logic error and exception in RS Gateway that would trigger a -2 response and an Internal Server
error HTTP response. This did not trigger a -15 transmission error response. |
| MTX-67988 | In RS Gateway, the first query of the external Event Repository of each tenant failed. |
| MTX-67733 |
SBA Gateway closed the connection with Kafka when there were messages to be sent to Kafka and
these messages were then written to PV (WTD) before SBA Gateway shutdown completely. Now, when
shutdown is triggered, SBA Gateway sends a GOAWAY frame to any new request received. If Kafka
Write-to-disk is enabled, then a new property gateway.external.kafka.logger.shutdownLoggingGraceSeconds is used to wait for any pending records to be
written to disk.Note: This changes how SBA Gateway interacts with the
SMF/PCF. During the graceful shutdown period, any established HTTP2 connections from the SMF/PCF to the SBA receive an HTTP2 GOAWAY frame. This should cause the
SMF/PCF to gracefully close the HTTP2 connection as specified by RFC 7540 6.8. |
| MTX-67719 | The SBA Gateway ping message did not re-establish a TCP connection that was broken in an ungraceful manner. |
| MTX-66637 | The multi-query REST API included error records. Now, it only returns results that have result 0 indicating that they are a good result. Error records are not included in the response. |
| MTX-64704 | The snmp-exporter pod was not collecting SNMP stats from Network Enabler because Network Enabler listened on the wrong port. |
| MTX-46991 | When a message was suspended pending processing outside the MATRIXX system (for MAP callouts, call initiation, or Pay Now processing) the round trip time to and from the external system was counted as latency queueing in the Charging Server. |
Applications
| Issue Number | Description |
|---|---|
| MTX-68914 | The configure_event_repository.py script was hard-coded to run on a specific engine pod. |
| MTX-68794 | During MATRIXX Backoffice Customer Tool bulk operations, setting an array field with a single value triggered an exception. Setting it with at least two values was correctly processed. |
| MTX-68605 | MATRIXX Backoffice Customer Tool treated LoginId and AccessId as
values were required to be numbers when searching. |
| MTX-67937 | In MATRIXX Backoffice Customer Tool, adding a device in the Subscription page failed. Now,
TrafficRouteData is set to the RouteId for the sub-domain selected (if one is selected). |
| MTX-67921 | There was an issue in the general search API for MATRIXX Backoffice Customer Tool where the secondary search would fail processing after a subscriber and device were rehomed. |
| MTX-67534 | Some icons were missing in MATRIXX Backoffice Customer Tool when first loading a page. |
| MTX-66405 | If one of the following configuration question values was 0, then no DIAMETER response was sent when the criteria in the question was met. A 5012 error was still
recorded and logged in the SNMP stats (viewable using the print_blade_stats.py
script).Now, stats are not updated if no DIAMETER response was sent. |
| MTX-64897 | When a bulk operation job was canceled, task information was deleted before the cancellation was successful. |