Fix OOM issue # 242

[pravinbhat]

What this PR does:
Fixes OOM exception when partition file present and numParts has a high value

Which issue(s) this PR fixes:
Fixes #242

Checklist:

• Automated Tests added/updated
• Documentation added/updated
• CLA Signed: DataStax CLA

[faizalrub-datastax]

Praveen, I see that the MAX_NUM_PARTS_FOR_PARTITION_FILE = 10. Does this mean always the numParts will be 10 when reading partition ranges from CSV file? There could be partition ranges that require more than 10 for large datasets.

[pravinbhat]

Praveen, I see that the MAX_NUM_PARTS_FOR_PARTITION_FILE = 10. Does this mean always the numParts will be 10 when reading partition ranges from CSV file? There could be partition ranges that require more than 10 for large datasets.

Good question @faizalrub-datastax. No, the num-parts value further splits each partition that is listed in the partition csv file. i.e. If the partition file has 100 partitions min/max lines & num-parts is set to 10, that will effectively convert to 100 * 10 = 1000 partitions. What was happening now is, when we run a default job with 10K numparts & then there are a few failures, let say 100. If you rerun this job, it will retry those 100 failed partition by split each of those partition by 10K, so effectively 100 * 10K = 1M partitions, which causes the OOM. Note that MAX_NUM_PARTS_FOR_PARTITION_FILE prop is used only when using a partition-file with the partitions listed. Ideally in such scenarios, the num-part should be set to 1 unless the failure was caused due to a large partition-range. If it is a large partition-range the max we may need to further split will not be more than 10. Hope this clarifies.