Scriveners in the 1861 census?

My great-great-grandmother Harriet Scrivener was born in Luton in 1851 (I have her birth certificate), daughter of William and Hannah (nee Hawkes or Hawker). She married Jesse Ephgrave in 1873. She appears in the 1851 census with her parents, living at Elizabeth Street, Luton, where she was born, and siblings Mary Ann (aged 14, a bonnet sewer) and Louisa, aged 2.  Father William is working as a Groom, aged 38 and born in Limbury, Beds, whilst mother Anna (or Hannah) is 37, a bonnet sewer, whose Bedfordshire birthplace is difficult to read.

The family is still in Luton at the time of the 1871 census, living at 23 Stuart Street. William (55) and Hannah (56) are both shown as born in Luton, Bedfordshire, and their occupations are the same as at the time of the 1851 census. Still at home are daughters Mary Ann (transcribed as Marguerite), Harriett and Sarah (aged 15), all described as hat sewers. 

So where are they at the time of the 1861 census? Their surname is of course open to much mis-transcription (one index of the 1871 shows them as Serwinor) but searches for various members of the family by first name and birth date/place only also fails to find them (so far). I'd be very pleased if anyone can find them ... although I know some parts of the census returns are missing, and this may just be one of them.