[cubestore] corrupt data error: File **.parquet doesn't exist in remote file system

[kevinleeTCA]

Problem

We keeps having this error msg on daily basis in our cubestore logs:

...

2025-01-30 15:17:46.715 | 2025-01-30T04:17:46.715Z ERROR [cubestore::queryplanner::query_executor] <pid:1> Error Query (126.880301ms):
2025-01-30 15:17:46.715 | 2025-01-30T04:17:46.715Z INFO  [cubestore::metastore] <pid:1> Deactivating table prod_pre_aggregations.gl_income_rollup20240101_kla32oge_di0ibpro_1jkd7n6 (#2427) due to corrupt data error: File 9165-ncfolk1k.parquet doesn't exist in remote file system

...

Related Cube.js schema

cube(`GeneralLedger_Income`, {
  sql: `${incomeSQL}
    where ${incomeSQLFilter}
    `,
  extends: GeneralLedger_BaseIncome,
  sqlAlias: 'glIncome',
  preAggregations: {
    rollup: {
      measures: [CUBE.incomeAmount, CUBE.count, CUBE.managementFeeIncomeAmount, CUBE.minDate],
      dimensions: [CUBE.transactionTaxCategoryId, CUBE.generalLedgerManagement, CUBE.accountOwner, CUBE.accountBookType],
      timeDimension: CUBE.createdAt,
      granularity: `day`,
      indexes: {
        idx: {
          columns: [ CUBE.accountOwner, CUBE.generalLedgerManagement],
        },
      },
      refresh_key: {
        every: `30 6 * * *`,
        timezone: `Australia/Sydney`,
        incremental: true,
        update_window: `60 days`,
      },
      partition_granularity: `month`,
      build_range_start: {
        sql: `SELECT '2020-08-11'::timestamp AT TIME ZONE 'utc'`,
      },
      build_range_end: {
        sql: `SELECT NOW()`,
      },
    },
  },
  measures: {},
  dimensions: {},
  segments: {},
  dataSource: `generalLedger`,
});

Related Query

2025-01-30 15:17:46.716 | 2025-01-30T04:17:46.715Z ERROR [cubestore::queryplanner::query_executor] <pid:1> Error Query Physical Plan (126.880301ms): GlobalLimit, n: 3000 |  
-- | -- | --
  |   | 2025-01-30 15:17:46.716 | Scan p_m__fee_tax_categories__rollup, source: CubeTable(index: p_m__fee_tax_categories_rollup_idx_h4fwcpsc_4l4cgk5j_1jjaiga:1346:[2648, 2684]:sort_on[p_m__fee_tax_categories__tax_category_id]), fields: * |  
  |   | 2025-01-30 15:17:46.716 | Scan groups_by_ma__g_b_m_rollup, source: CubeTable(index: groups_by_ma_g_b_m_rollup_mg_idx_xde2o1oi_sdfcdeov_1jpls5j:16773:[32798]:sort_on[groups_by_ma__management_ailorn]), fields: [groups_by_ma__management_ailorn] |  
  |   | 2025-01-30 15:17:46.716 | Scan prod_pre_aggregations.gl_income_rollup20240601_y1vcbtr3_tvbyjwvf_1jplegd, source: CubeTable(index: gl_income_rollup_idx_y1vcbtr3_tvbyjwvf_1jplegd:16738:[32734, 32749]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.716 | Scan prod_pre_aggregations.gl_income_rollup20240501_agnnisxe_is1zsk0r_1jplegd, source: CubeTable(index: gl_income_rollup_idx_agnnisxe_is1zsk0r_1jplegd:16736:[32732, 32748]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.716 | Scan prod_pre_aggregations.gl_income_rollup20240401_4rr0etsz_50d11yiv_1jplesf, source: CubeTable(index: gl_income_rollup_idx_4rr0etsz_50d11yiv_1jplesf:16740:[32740, 32751]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20240301_msq2aw0n_ykwidr0u_1jkd5ic, source: CubeTable(index: gl_income_rollup_idx_msq2aw0n_ykwidr0u_1jkd5ic:4619:[9073, 9096]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20240201_tjjj3bbi_aujhlb1e_1jkd63j, source: CubeTable(index: gl_income_rollup_idx_tjjj3bbi_aujhlb1e_1jkd63j:4629:[9091, 9112]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20240101_kla32oge_di0ibpro_1jkd7n6, source: CubeTable(index: gl_income_rollup_idx_kla32oge_di0ibpro_1jkd7n6:4657:[9143, 9165]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20231201_eevk4axq_angupusd_1jpls6k, source: CubeTable(index: gl_income_rollup_idx_eevk4axq_angupusd_1jpls6k:16781:[32810, 32822]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20231101_m0i0v5sg_na1ecvvu_1jpls6k, source: CubeTable(index: gl_income_rollup_idx_m0i0v5sg_na1ecvvu_1jpls6k:16779:[32808, 32823]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20231001_y4xyjrqr_jg1k5lim_1jplslq, source: CubeTable(index: gl_income_rollup_idx_y4xyjrqr_jg1k5lim_1jplslq:16783:[32814, 32828]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20230901_ans5yvsu_05ydwpmt_1jpls7t, source: CubeTable(index: gl_income_rollup_idx_ans5yvsu_05ydwpmt_1jpls7t:16777:[32806, 32818]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20230801_30cmavvy_zpgdclsh_1jplt4k, source: CubeTable(index: gl_income_rollup_idx_30cmavvy_zpgdclsh_1jplt4k:16787:[32826, 32833]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Scan prod_pre_aggregations.gl_income_rollup20230701_zgkopi5w_wflmgv1y_1jpls7t, source: CubeTable(index: gl_income_rollup_idx_zgkopi5w_wflmgv1y_1jpls7t:16775:[32804, 32816]:sort_on[gl_income__account_owner, gl_income__general_ledger_management]), fields: [gl_income__account_book_type, gl_income__account_owner, gl_income__general_ledger_management, gl_income__transaction_tax_category_id, gl_income__created_at_day, gl_income__count, gl_income__income_amount, gl_income__management_fee_income_amount] |  
  |   | 2025-01-30 15:17:46.715 | Union |  
  |   | 2025-01-30 15:17:46.715 | Scan p_m__teams__p_m_teams_rollup, source: CubeTable(index: p_m__teams_p_m_teams_rollup_mg_idx_lpogm2te_gablnqqs_1jpls5i:16771:[32796, 32813]), fields: [p_m__teams__legal_entity_ailorn, p_m__teams__management_ailorn, p_m__teams__organisation_id, p_m__teams__property_type] |  
  |   | 2025-01-30 15:17:46.715 | Filter |  
  |   | 2025-01-30 15:17:46.715 | Join on: [#p_m__teams__p_m_teams_rollup.p_m__teams__legal_entity_ailorn = #gl_income__rollup.gl_income__account_owner, #p_m__teams__p_m_teams_rollup.p_m__teams__management_ailorn = #gl_income__rollup.gl_income__general_ledger_management] |  
  |   | 2025-01-30 15:17:46.715 | Filter |  
  |   | 2025-01-30 15:17:46.715 | Join on: [#p_m__teams__p_m_teams_rollup.p_m__teams__management_ailorn = #groups_by_ma__g_b_m_rollup.groups_by_ma__management_ailorn] |  
  |   | 2025-01-30 15:17:46.715 | Join on: [#gl_income__rollup.gl_income__transaction_tax_category_id = #p_m__fee_tax_categories__rollup.p_m__fee_tax_categories__tax_category_id] |  
  |   | 2025-01-30 15:17:46.715 | ClusterSend, indices: [[16771], [16775, 16787, 16777, 16783, 16779, 16781, 4657, 4629, 4619, 16740, 16736, 16738], [16773], [1346]] |  
  |   | 2025-01-30 15:17:46.715 | Aggregate |  
  |   | 2025-01-30 15:17:46.715 | Projection, [gl_income__transaction_tax_category_id, p_m__fee_tax_categories__fee_tax_category_name, gl_income__income_amount, gl_income__management_fee_income_amount, gl_income__count] |  
  |   | 2025-01-30 15:17:46.715 | Sort |  
  |   | 2025-01-30 15:17:46.715 | Limit