Here is an alternative method based on Asynchronous calls using
Async...Await[
^].
Async...Await uses the ThreadPool to manage the multitasking. The number of threads executed simultaneously is dependant on the CPU and the number of cores. If you want to change the default setting, the following documentation will explain how:
ThreadPool.SetMinThreads(Int32, Int32) Method (System.Threading) | Microsoft Docs[
^]
The
WebClient[
^] class has a
DownloadStringTaskAsync[
^] method that we can use. We can force the method to run off the UI thread by calling
ConfigureAwait(false)[
^].
To help us visualize how this work, let us wrap the call in a method:
private static async Task<Tuple<string, string>> GetUrlAsync(Uri url)
{
Console.WriteLine("thisIsAsyncStart: "
+ Thread.CurrentThread.ManagedThreadId);
var result = await new WebClient()
.DownloadStringTaskAsync(url).ConfigureAwait(false);
Console.WriteLine("thisIsAsyncEnd: " + Thread.CurrentThread.ManagedThreadId);
return new Tuple<string, string>(url.ToString(), result);
}
Next, we need a few URIs to call:
private static List<Uri> urls = new List<Uri>()
{
new Uri("https://www.codeproject.com", UriKind.Absolute),
new Uri("https://apple.com", UriKind.Absolute),
new Uri("https://xamarin.com", UriKind.Absolute),
new Uri("https://nokia.com", UriKind.Absolute),
new Uri("https://samsung.com", UriKind.Absolute),
new Uri("https://www.microsoft.com", UriKind.Absolute),
new Uri("https://www.codeproject.com", UriKind.Absolute),
new Uri("https://apple.com", UriKind.Absolute),
new Uri("https://xamarin.com", UriKind.Absolute),
new Uri("https://nokia.com", UriKind.Absolute),
new Uri("https://samsung.com", UriKind.Absolute),
new Uri("https://www.microsoft.com", UriKind.Absolute),
new Uri("https://www.codeproject.com", UriKind.Absolute),
new Uri("https://apple.com", UriKind.Absolute),
new Uri("https://xamarin.com", UriKind.Absolute),
new Uri("https://nokia.com", UriKind.Absolute),
new Uri("https://samsung.com", UriKind.Absolute),
new Uri("https://www.microsoft.com", UriKind.Absolute),
new Uri("https://www.codeproject.com", UriKind.Absolute),
new Uri("https://apple.com", UriKind.Absolute),
new Uri("https://xamarin.com", UriKind.Absolute),
new Uri("https://nokia.com", UriKind.Absolute),
new Uri("https://samsung.com", UriKind.Absolute),
new Uri("https://www.microsoft.com", UriKind.Absolute),
new Uri("https://www.codeproject.com", UriKind.Absolute),
new Uri("https://apple.com", UriKind.Absolute),
new Uri("https://xamarin.com", UriKind.Absolute),
new Uri("https://nokia.com", UriKind.Absolute),
new Uri("https://samsung.com", UriKind.Absolute),
new Uri("https://www.microsoft.com", UriKind.Absolute),
};
Now there are two way that I will demonstrate managing multiple tasks at once.
1.
WhenAll[
^] -
Creates a task that will complete when all of the Task objects in an array have completed
private static async Task RunWhenAllAsync()
{
var tasks = new List<Task<Tuple<string, string>>>();
foreach (var url in urls)
{
tasks.Add(GetUrlAsync(url));
}
await Task.WhenAll(tasks.ToArray());
foreach (var task in tasks)
{
var result = task.Result;
Console.WriteLine($"id: {task.Id} | url {task.Result.Item1} > length: {task.Result.Item2.Length}");
}
}
Output:
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncEnd: 13
thisIsAsyncEnd: 11
thisIsAsyncEnd: 13
thisIsAsyncEnd: 13
thisIsAsyncEnd: 11
thisIsAsyncEnd: 16
thisIsAsyncEnd: 11
thisIsAsyncEnd: 12
thisIsAsyncEnd: 16
thisIsAsyncEnd: 15
thisIsAsyncEnd: 12
thisIsAsyncEnd: 12
thisIsAsyncEnd: 13
thisIsAsyncEnd: 11
thisIsAsyncEnd: 16
thisIsAsyncEnd: 11
thisIsAsyncEnd: 11
thisIsAsyncEnd: 16
thisIsAsyncEnd: 14
thisIsAsyncEnd: 14
thisIsAsyncEnd: 13
thisIsAsyncEnd: 12
thisIsAsyncEnd: 15
thisIsAsyncEnd: 14
thisIsAsyncEnd: 12
thisIsAsyncEnd: 13
thisIsAsyncEnd: 14
thisIsAsyncEnd: 11
thisIsAsyncEnd: 15
thisIsAsyncEnd: 11
id: 1 | url https://www.codeproject.com/ > length: 96485
id: 3 | url https://apple.com/ > length: 53646
id: 5 | url https://xamarin.com/ > length: 149548
id: 7 | url https://nokia.com/ > length: 73246
id: 9 | url https://samsung.com/ > length: 106147
id: 11 | url https://www.microsoft.com/ > length: 1020
id: 13 | url https://www.codeproject.com/ > length: 96479
id: 15 | url https://apple.com/ > length: 53646
id: 17 | url https://xamarin.com/ > length: 149548
id: 19 | url https://nokia.com/ > length: 73246
id: 21 | url https://samsung.com/ > length: 106147
id: 23 | url https://www.microsoft.com/ > length: 1020
id: 25 | url https://www.codeproject.com/ > length: 96485
id: 27 | url https://apple.com/ > length: 53646
id: 29 | url https://xamarin.com/ > length: 149548
id: 31 | url https://nokia.com/ > length: 73246
id: 33 | url https://samsung.com/ > length: 106147
id: 35 | url https://www.microsoft.com/ > length: 1020
id: 37 | url https://www.codeproject.com/ > length: 96485
id: 39 | url https://apple.com/ > length: 53646
id: 41 | url https://xamarin.com/ > length: 149548
id: 43 | url https://nokia.com/ > length: 73246
id: 45 | url https://samsung.com/ > length: 106147
id: 47 | url https://www.microsoft.com/ > length: 1020
id: 49 | url https://www.codeproject.com/ > length: 96485
id: 51 | url https://apple.com/ > length: 53646
id: 53 | url https://xamarin.com/ > length: 149548
id: 55 | url https://nokia.com/ > length: 73246
id: 57 | url https://samsung.com/ > length: 106147
id: 59 | url https://www.microsoft.com/ > length: 1020
Benefits:
- code is very simple
- the UI thread won't be blocked
- the tasks run in async on seperate threads to the UI
- the results are in the order the tasks were created, not the order they completed
Disadvantages:
- execution will wait until all tasks are completed
- the user won't receive any progress feedback
2.
WaitAny[
^] -
Waits for any of the provided Task objects to complete execution.
private static readonly object lockObj = new object();
private static async Task RunWaitAnyAsync()
{
var tasks = new List<Task<Tuple<string, string>>>();
foreach (var url in urls)
{
tasks.Add(GetUrlAsync(url));
}
while (tasks.Count > 0)
{
var index = Task.WaitAny(tasks.ToArray());
var task = tasks[index];
if (task.IsCompleted)
{
var result = task.Result;
Console.WriteLine($"id: {task.Id} | url {task.Result.Item1} > length: {task.Result.Item2.Length}");
}
else if (task.IsFaulted)
{
}
lock (lockObj) tasks.Remove(task);
}
}
Output:
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncStart: 1
thisIsAsyncEnd: 12
thisIsAsyncEnd: 13
id: 11 | url https://www.microsoft.com/ > length: 1020
id: 23 | url https://www.microsoft.com/ > length: 1020
thisIsAsyncEnd: 13
id: 35 | url https://www.microsoft.com/ > length: 1020
thisIsAsyncEnd: 13
id: 47 | url https://www.microsoft.com/ > length: 1020
thisIsAsyncEnd: 11
id: 59 | url https://www.microsoft.com/ > length: 1020
thisIsAsyncEnd: 13
id: 3 | url https://apple.com/ > length: 53646
thisIsAsyncEnd: 16
id: 15 | url https://apple.com/ > length: 53646
thisIsAsyncEnd: 15
id: 27 | url https://apple.com/ > length: 53646
thisIsAsyncEnd: 15
id: 7 | url https://nokia.com/ > length: 73246
thisIsAsyncEnd: 12
id: 39 | url https://apple.com/ > length: 53646
thisIsAsyncEnd: 13
thisIsAsyncEnd: 16
id: 9 | url https://samsung.com/ > length: 106147
id: 21 | url https://samsung.com/ > length: 106147
thisIsAsyncEnd: 13
id: 31 | url https://nokia.com/ > length: 73247
thisIsAsyncEnd: 14
id: 19 | url https://nokia.com/ > length: 73246
thisIsAsyncEnd: 15
id: 55 | url https://nokia.com/ > length: 73247
thisIsAsyncEnd: 17
id: 45 | url https://samsung.com/ > length: 106148
thisIsAsyncEnd: 16
id: 33 | url https://samsung.com/ > length: 106148
thisIsAsyncEnd: 17
id: 57 | url https://samsung.com/ > length: 106147
thisIsAsyncEnd: 16
id: 43 | url https://nokia.com/ > length: 73247
thisIsAsyncEnd: 14
id: 51 | url https://apple.com/ > length: 53646
thisIsAsyncEnd: 11
id: 41 | url https://xamarin.com/ > length: 149548
thisIsAsyncEnd: 12
id: 17 | url https://xamarin.com/ > length: 149548
thisIsAsyncEnd: 16
id: 53 | url https://xamarin.com/ > length: 149548
thisIsAsyncEnd: 15
id: 5 | url https://xamarin.com/ > length: 149548
thisIsAsyncEnd: 15
id: 1 | url https://www.codeproject.com/ > length: 96485
thisIsAsyncEnd: 17
id: 29 | url https://xamarin.com/ > length: 149548
thisIsAsyncEnd: 15
id: 25 | url https://www.codeproject.com/ > length: 96479
thisIsAsyncEnd: 17
id: 49 | url https://www.codeproject.com/ > length: 96479
thisIsAsyncEnd: 11
id: 13 | url https://www.codeproject.com/ > length: 96485
thisIsAsyncEnd: 16
id: 37 | url https://www.codeproject.com/ > length: 96485
Benefits:
- the UI thread won't be blocked
- the tasks run in async on seperate threads to the UI
- the results are in the order of completion, not the order they were executed
- Feedback is imediate as a task completes giving the user progress feedback
Disadvantages:
NOTE: If you have hundreds or thousands of pages that you want to pull down, I have a better ramp up function for not overloading the Threadpool system & CPU.