Is there a rule of thumb as to how high a box should be for the step up in relation to a lifters height?
I usually look at the angle of the thighs and try to get the box high enough to approximate legal squat depth. Like you, I'm quite a newbie on the movement and some reps tend to look quite ugly as I struggle to keep the balance. The fairly high stack of boxes is probably a contributing factor, getting the box lower would probably help in that regard.
I have a hate-love relationship with the step-up: I hate the low weights I can handle, but love what it ought to do for my weak quads. I've also done my share of lunges, but find the step-up much more controllable. I spent my youth doing martial arts, but you couldn't tell from seeing me do these...
Looking forward to seeing more suggestions.