Some info from findmypast.com - hope it's not repeating or confusing.
1851 census - Canonbury Square, Islington, London
William Rogerson, age 52 (1799) Elastic Fabric Merchant, born Ireland
Annie I Rogerson, age 44 (1807) wife, born Ireland
Ellen M Rogerson, age 26 (1825) daughter, born Ireland
Mary A Rogerson, age 24 (1827) daughter, born Ireland
Emily F Rogerson, age 20 (1831) daughter, born Ireland
Louisa I Rogerson, age 18 (1833) daughter, born Ireland
Adelaide Rogerson, age 17 (1834) daughter, born Ireland
Florence I Rogerson, age 11 (1840) daughter, scholar, born Ireland
William T Rogerson, age 9 (1842) son, scholar, born Ireland
James T Rogerson, age 6 (1845) son, scholar, born Ireland
George Rogerson, age 4 (1847) son, norn Liverpool
There is also Ellen McCarthy, age 33 (1818) servant, born Ireland
That would suggest that the family came over from Ireland sometime between 1845 and 1847.
1861 census - 23 Church road, Hackney, London
William B Rogerson, age 62 (1799) retired merchant, born Ireland
Ann J Rogerson, age 54 (1807) wife, born Ireland
Emily F Rogerson, age 29 (1832) governess, born Ireland
Florence Rogerson, age 20 (1841) governess, born Ireland
George N Rogerson, age 14 (1847) clerk in a railway office, born Liverpool
There is also Henry Newgam, age 28 (1833) boarder, chemist, born Lincolnshire and
Catherine Connor, age 35 (1826) general servant, born Ireland
1871 censue - no trace of William Rogerson but could, from your note above, have been back in Ireland.
1871 census - Middle Street, South Place, Camberwell, London
Ann Rogerson, age 62 (1809) lodger, born Ireland ... could be the same Ann Rogerson above.
1881 census - Carnarvon Terrace, Llangefni, Anglesey
Thoams Davies, age 63 (1818) formerly farm servanmt, born Anglesey
Mary Rogerson, age 35 (1846) widow, lodger, Nurse (SMS), born Llangwyllog, Anglesey
Roger Rogerson, age 6 (1875) lodger, scholar, born Llangefni
Mary A Rogerson, age 7 (1874) lodger, scholar, born Liverpool
William B Rogerson, age 3 (1878) lodger, born Bangor, Caernarvonshire
There is also Catherine Jones, age 31 (1850) lodger, general domestic servant, born Llwngwyllog
There is also abirth in 1877 of William Bell Rogerson, 2nd quarter, on Bangor - mother's maiden mame Jones.
Noting your comment earlier about being rich to live in St George Hanover Square, London ... my maternal grandfather (George Roper) was born there in 1896 and the 1901 census shows him as living in Eaton Mews South, he was one of seven children. His father (also George Roper) was a groom and looking at the neighbouring properties there were a number of grooms and stable workers - I believe the street comprised stables. Certainly my family were not rich!!! The street now comprises some very exclusive and expensive apartments, and google street map won't take you down there!