Notifications

380 views

Description

Some of the customers with big billing data are complaining that the current Azure billing process is too slow. It turns out that the size of data is too big for the transformation process.

Symptoms

  • The Billing Job or the Billing Schedule is very slow to complete.
  • The Billing job seems to be stuck or hung at some point and no progress in the background.
  • The Billing job unable to complete or no data available in transform history.

Example

  • The schedule shows that 2019/02 import succeeded but 2019/01 import is stuck.
  • Workflow shows that the process is stuck during transformation. which means parsing is done and all billing data is in import table - sn_cmp_imp_azure_cost
  • # of records in sn_cmp_imp_azure_cost for 2019/01 : 1,271,202
  • According to transform history, two transforms have been stuck since 02-19-2019. These two transforms are causing the issue as per the above screenshot.

 

Steps to Reproduce

  1. Login to the instance 
  2. Cloud Management > Cloud Admin Portal > Analyze > Billing 
  3. Billing Schedules > Verify the billing schedule in the hung state > Open the record 
  4. Observe either the Billing job is either hung or running very slow than expected

Workaround

There are multiple updates with different issues on billing with Azure and AWS, and this is a part of the parent issue, hence consolidated all the fixes and released the Workaround along with updates.

Please refer "Mismatch in AWS / Azure and CMP Dashboard billing data" for the updates to implement as a workaround

The fix for this issue contains: 

  • Added new indexes for fast retrieval
  • Created AzureBillingCSVRollupProcessor. This script will be exected to populate monthly rolled-up data.
  • Enabled the new processor by replacing the value (sn_cmp.AzureBillingCSVRollupProcessor()) for provider: Azure and key: billing_processor_constructor
  • Created a new property : sn_cmp.billing.use_tag_values_for_pivot for tracking tag_values change. The default is "false", Modified CmpBillingCSVProcessor & AzureBillingCSVProcessor 

 

IMP: When applying the attached updateset, also make sure that this property should be set to 0 - sn_cmp.billing.update_threshold = 0


Related Problem: PRB1331941

Seen In

SR - IRM - Audit Management - New York 2019 Q3
SR - IRM - GRC Profiles - Madrid 2019 Q2
SR - IRM - GRC Workbench - New York 2019 Q3
SR - IRM - PA Premium Integration - New York 2019 Q3
SR - IRM - Policy and Compliance - Madrid 2019 Q2
SR - IRM - Risk Management - New York 2019 Q3
SR - Security - Integration Framework - Madrid 2019 Q2
SR - Security - Support Common - Madrid 2019 Q2
SR - Security - Support Orchestration - Madrid 2019 Q2
SR - SIR - Security Incident Response - Madrid 2019 Q2
SR - SIR - Security Incident Response PA Content - New York 2019 Q3
SR - SIR - Store SecOps Setup Assistant - Madrid 2019 Q2
SR - SIR - Store Threat Core - Madrid 2019 Q2
SR - SIR - Store Trusted Security Circles Client - New York 2019 Q3
SR - VR - Vulnerability Response - New York 2019 Q3
SR - VR - Vulnerability Response PA Content - Madrid 2019 Q2

Fixed In

London Patch 8
Madrid Patch 3
New York

Associated Community Threads

There is no data to report.

Article Information

Last Updated:2019-10-16 03:54:39
Published:2019-07-07