Looks to me the pattern is moving twards the center line of the axle with the less shim you install.
Just as it should.
ADD MORE SHIMS!! LOTS OF THEM!!!!!!
You are not making big enough changes and going the wrong way.
Just my .02 and what do I know.