Yes, median (pitch diameter) is the point where the strength comes in to play. And yes, that is assuming alike materials. I know where you are going by saying if you go to a course thread that you will get deeper into the aluminum. But, don't forget, you are using the aluminum "as the nut". This means that you want your pitch diameter to be larger, as in fine thread.
If you were using a steel nut and aluminum bolt, I would agree with you more.