I tried rotating the map a little and realise that the rotation is the exact opposite of what I want. When I move the mouse to the rightt, to me I'm dragging the cloth to the right but it turns the other way.

I realised that it depends on where your mouse cursor is. When you're in the middle of the map, it does weird things because to me I'm in the lower part but to the computer, I'm in the upper part. Rotation should maybe be slower if you're near the screen center? Also, if you move slightly up  or down and release the mouse, you end up in the other half of the screen, and suddenly moving still in the same left-right direction moves your camera the other way.