pilot who recently retired come to me for advice as to how to become an HAA pilot. He had flown Reserve helicopters for 30 years while flying for the airlines. Told him to go get recurrent in whatever helicopter he could afford, to include an IFR currency, then, go afield and get some HAA time, then apply for whatever job he wanted withing the company he preferred. He did just that, and got what he wanted.
Point is, like you mentioned, keep slugging, you'll get there, especially in HAA.