Sadly I have to agree with Hugo and Helig about the 1861 census and the three siblings.
Also, apologies if any of the below has been found and posted before.
I thought I would try the 1881 census and found Robert and his Sarah Ellen living at Seaview Terrace. I also found " an" Edward and Lydia possibly the witnesses to their wedding living in Pool street. That Lydia was born in Oswestry in 1842 and if it's the correct one she is older than Robert who was born in 1855. I'm pretty certain it's the right Robert but not quite so with the Lydia(particularly as she would have retained her maiden name on marriage). I have no time today to do any more research but I thought I'd just post them anyway and let others comment further.... ( keep scrolling down!)
Robert Williams
England and Wales Census, 1881
Name: Robert Williams
Event Type: Census
Event Date: 1881
Event Place: Conway, Caernarvonshire, Wales
Registration District: Conway
Residence Note: Sea View Ter
Gender: Male
Age: 26
Marital Status: Married
Occupation: Labourer (Gen)
Relationship to Head of Household: Head
Birth Year (Estimated): 1855
Birthplace: Llangelynin, Caernarvonshire, Wales
Page Number: 28
Registration Number: RG11
Piece/Folio: 5581/28
Affiliate Record Type: Household
Household Role Sex Age Birthplace
Robert Williams Head Male 26 Llangelynin, Caernarvonshire, Wales
Sarah E Williams Wife Female 25 Llangerniew, Denbighshire, Wales
Lydia Williams
England and Wales Census, 1881
Name: Lydia Williams
Event Type: Census
Event Date: 1881
Event Place: Conway, Caernarvonshire, Wales
Registration District: Conway
Residence Note: Pool Lane
Gender: Female
Age: 40
Marital Status: Married
Relationship to Head of Household: Wife
Birth Year (Estimated): 1841
Birthplace: Oswestry, Shropshire, England
Page Number: 46
Registration Number: RG11
Piece/Folio: 5581/37
Affiliate Record Type: Household
Household Role Sex Age Birthplace
Edward Williams Head Male 42 Conway, Caernarvonshire, Wales
Lydia Williams Wife Female 40 Oswestry, Shropshire, England
Mary Williams Daughter Female 13 Conway, Caernarvonshire, Wales
Ann Williams Daughter Female 11 Conway, Caernarvonshire, Wales
Edward Smith Williams Son Male 9 Conway, Caernarvonshire, Wales
John Robert Williams Son Male 7 Conway, Caernarvonshire, Wales
William James Williams Son Male 4 Conway, Caernarvonshire, Wales