Your Aunt may well be correct and the Park Hotel may be just a coincidence and the white house is somewhere else. When you said that one family of Rogerson lived at the top and one at the bottom, my initial thought was that the top one lived in Bryn Pydew which is a small village at the top of a hill above where Pen Sarn Farm was.
But as yet we haven't established where exactly the top Rogerson family lived
At one time Bryn Pydew did have a shop, pub and a number of large white houses nearby but I don't know when the shop and pub closed. There were two pubs there, the Red Lion which was run as a pub in the 19th Century and the old Swan but I don't know when that ceased as a pub, before becoming a shop.
Any photos of property may help to trace locations and if you post on here someone may be able to recognise the house.