I'd suggest getting your (on the picture left) shoulder bell closer to the shoulder bridge. This isn't required for basic approval, but I've found that it improves the overall appearance of the shoulder bells to minimise that gap.
Another gap that you might want to minimise is the gap between the bicep armor and the shoulder bell, possibly by taking away a bit of the armor on the shoulder bell in roughly the shape that MaskedVengeance already lined out. Another thing that might help would be taking away the return edge. Justjoseph63 has a great post on how much return edge you can take away:
As for cutting off too much, simply try to cut a few millimetres at a time and do another test fit, then cut again and so on. Just don't forget to compare your armor to the references.
I hope this might have helped you!