The biggest reason that keep us away from going down to low body fat level.

When you dieting down you are not only losing muscle but also fats & water underneath the muscle and skin. You can’t expect losing weight without losing any weight (size).

The good news is when you reach low body fat level (anything below 10%).
You will actually look bigger than you were in bulk season. I called this muscle illusion.
If are looking very small, then probably you are over doing it or you have to put more work in gaining muscle mass first.

source