I think you have stumbled into the biggest problem with working out if you are Aromantic, Asexual, or both. 'How do you find the absence of something you never had?' It is really difficult and I have my days where I wonder if I'm faking all of it or if it's all just some late blooming/hormone issue or something. What I can say now is that I have less of those days and honestly that's all I can hope for.
One thing that helped me solidify some of it was a book called Loveless by Alice Oseman. There were some great descriptions and advice in there which helped me realise AroAce was right for me however if were honest this is probably just a me thing and I can in now way guarantee it will help.
I will reiterate what others have said as well, look into Aesthetic attraction to see if that matches what you feel. Also Demiromantic may describe what you were having with your childhood friend. However if the labels are what causing you issues maybe try being label-less for a bit. I'm sure you know this but you don't have to label anything especially if it is making you anxious. If you have any more questions feel free to ask here or Private Message me if you have any more questions you would prefer in a less social sphere for any reason at all.