Lots of confusing stuff about concurrency and parallelism

[colinleach]

N.B. This is not a PR for formal review!

Think of it as:

• a wildly incomplete draft that I will keep adding to
• an early heads up that "Oh $!**, we're doomed" on this as a concept

Even with what we have so far, I think that you can see that this is not a viable single concept. More like a syllabus in its own right.

It's not immediately obvious how we split it up. The various bits tend to work together, quite closely interwoven.

I'll keep writing (and correcting, as I try to understand it better).

I think it would be good to think about exercises, both concept and practice.

• You already suggested one potential concept exercise (which I need to search for). What subset of the topics would this need?
• I'll look for anything relevant on other tracks, but I'm drawing a blank so far. I thought Go would have covered this, given that goroutines and channels are its main selling point, but apparently they didn't get to it in the syllabus.
• It seems worth a forum post to ask for help from other maintainers. Maybe somebody discussed this before, even if it wasn't merged?
• We should think about bank-balance and parallel-letter-frequency to see how these practice exercises might be implemented. The first uses locks and/or atomics, the latter needs a split-run-reduce (aka scatter-gather) algorithm.

[depial]

I'm also currently finding out how big of a concept this really is, in the context of GPU programming 😅
I found that thinking in terms of sequential programming is like thinking in English, where word order matters, while thinking in parallel programming is like thinking in Hungarian or Latin, where word order is unimportant. These have completely different grammar and require different methods and tactics to complete the same problem. It's hard to wrap my mind around it, but has given me quite an interesting new perspective on the world.

I think that you can see that this is not a viable single concept.

I agree this should likely be split up. My initial thought is to break it up into (at least) two parts:

• Tools: tasks / channels / etc
• Methods: scheduling, avoiding data races, etc. This could include topics on locks, atomics, reductions etc.

I had initially envisioned having a separate concept on channels, since they are useful outside of multithreading, namely for collecting/iterating graphs. They are the equivalent to Python's yield in this respect. I'm just not sure if there's enough in that for a full concept or not. That said, it could be nice to give students some exposure to them before diving into the multithreading aspect.

You already suggested one potential concept exercise (which I need to search for). What subset of the topics would this need?

I thought I had opened a PR with this idea on Combinatorics.jl, but now I remember that I was putting that off until the more basic updates were accepted. It appears they don't have reviewers, so who knows if/when that will happen. When I think about it, it might be too heavy for a concept exercise because it is based an AbstractArray interface which uses an algorithm that can produce the nth lexicographical permutation (i.e. just give nth_permutation(original_permutation, n) and it spits out the nth lexicographical perm). I used the AbstractArrays interface to defined getindex(A, i::Int) as nth_permutation(A, i::Int), which then allows for multithreading. What I had thought of doing was providing nth_permutation as boilerplate and building something around that. The fun part of this is actually the interface, so I think this might better be left for a practice exercise for that concept.

We should think about bank-balance and parallel-letter-frequency to see how these practice exercises might be implemented.

I think this is a good way forward. If we were going to do the split above, it feels like these two would fall into the methods section, but I guess it just depends what we decide are tools and what are methods :D

[colinleach]

I'll keep adding to this document, at least for the next few days:

• It will give us an overview of what we might need to cover.
• Chopping up and recycling content is a lot easier than writing it from scratch.
• I need this for my own use!

Not much progress on finding relevant exercises on other tracks.

• elixir/rpn-calculator-inspection covers Elixir processes, somewhat like Julia tasks.
• javascript/translation-service is about promises, which aren't directly relevant to us.

Neither is ideal for our needs, but may inspire some ideas.

[colinleach]

I think I've gone about as far as I can at the moment. The main gaps are to do with channels, where I need to go away and practice with some real code to understand them better. Also, the distributed stuff is mostly TODO, but that's off-topic in any case.

It may be time to try implementing bank-account and parallel-letter-frequency. I'll take a look at those over the next few days, with <100% confidence of success.

[colinleach]

There is now a Forum discussion with several interesting suggestions.

I'll keep working on the two practice exercises. We also need to look more closely at x86/opening-night, which tackles atomic operations from a super-low level. Maybe something we could adapt: a compare and contrast with both locks and atomics in the same exercise???

@BethanyG posted a long list of links, which will need more time to look through properly. At a minimum, there is stuff in there which is well worth reading for my own education, and maybe some ideas for exercises. It also jogged my memory that I have a print copy of "C# Concurrency" on my bookshelf, which I should take a closer look at. Also "Parallel and High Performance Computing", though that may be more biased to big-hardware applications: the lead author works at Los Alamos.