The Gomme then built two buildings: a house and a chapel. Precisely where these two stood is a matter for argument and conjecture. It has been said that when creating her chapel, she repaired the Hermit’s building and enlarged it, and that part of this enlarged oratory is still preserved in All Saints’ church. Evidence for this theory lies in the presence in the south-west corner of the chancel of a respond-like projection with fragments of a Norman string-course dating from around 1150: it is argued that this archway is not a doorway or a window, therefore it must have opened into a chamber of some sort, on the site of the present south aisle of the church, and that this chamber was the Hermit's oratory. The rest of the church was added on, anything up to a century later. None of this is proven, however, and the stone under dispute may quite simply have been a random piece used in the re-build in around 1250.

If All Saints’ church is not necessarily on the site of the Gomme’s chapel, her house is of similarly uncertain site. One of the earliest investigators into Dale Abbey’s history said that her house was ‘a small distance southward’ of her chapel: if her chapel was on the same site as All Saints’ church, this would mean that her house was very close to the sandstone escarpment in which the Hermitage is found, and a later writer doubted that chapel and house were so close together. Another theory, mentioned in the Chronicle, is that her house lay under a pond, known as Roger d’Alesby’s pond, which is still visible, if a little silted up, to the west of the lane which comes to a stop just to the north of All Saints’ church: the Chronicle says that worked stones of the Gomme’s house were found in the making of this pond. A third theory is that the house was sited in the field to the south-east of the Abbey site, where, according to a Victorian writer, there used to be a tanyard: this field presents many humps and hollows and has never been fully investigated.

The Hermit seems to have ended his days in Depedale, respected for his piety, and eventually found dead in his cell (be it cave or chapel) by a pilgrim.
The Hermitage seems to have been enlarged in the 18th century by one Sir Robert Burdett, who entertained in it.  It was occupied at the beginning of the 19th century by a couple whose house was being repaired:  the wife bore a son while there, and a stocking frame was set up. In Victorian times a Mr Chandler, a former steward of the estate, gave permission to a William McConnell to fit a door and windows to the cave, to enable him to extort a fee from visitors for its exhibition; the writer who recorded this event commented that ‘fortunately the man died before he could accomplish his design’!

The Gomme’s interest in the Hermit and in Depedale were to have major ramifications for the future of the valley, not least for her son Richard, who became a canon and served here, possibly as chaplain in the little church, which became a chantry chapel for the de Grendon family.