Stop fighting the drywall, and snap in some Toggler wall anchors. A must have when installing floating shelf hardware over drywall.