we have a public async method with the following code in it
int page = 1;
int size = 50;
var activeDocuments = await GetDocuments(page, size);
while (activeDocuments .Count > 0)
{
foreach (var document in activeDocuments)
{
document.Points = await CalculatePoints(document, settings);
updatedDocument = await _feedRepository.UpdateFeedAsync(document);
}
page += 1;
activeDocuments = await GetDocuments(page, size);
}
But the problem we are facing is that records are being missed from the iteration. Suppose we have 600+ records and above code only updates around 300.
Seems like an issue with the parallel processing but unable to get the hint of it.
Any help would highly be appreciated.
Note: We are using Hangfire and the code above is a background job enqueued via Hangfire.
EDIT: [Solution] Seems like its the mongo issue with indexes, although I created an index on CreatedAt(Datetimeoffset) field but in vain. While If I dont sort the collection (sorted automatically by default index i.e. _id), it works fine.
What I have tried:
Looked into different articles on google related to foreach implementation w.r.t multi-threading/parallel processing etc.