It was a memorial to a beloved spouse and a happy marriage - as the tablet reveals, it 'was undertaken for the purpose of relieving a mind afflicted by the loss of a most excellent wife'. But it was also intended as an ornament to be seen from his house, and a place from which to enjoy the spectacular views.
Jones effectively acted as his own clerk of works and all the details of the Castle’s construction - building materials and the craftsmen's wages - survive in his handwritten Account Book. This gives a vivid picture of the complexities of a building project. Good quality freestone for the copings and parapets could not be obtained locally and so had to be brought from Bath. Travelling via Bristol, barges, sloops and finally wagons were all necessary to get this stone to Clytha.
In 1791 work was well advanced as the ledger starts to show payments to the 'plaisterer' and for a specialist joiner from Worcester. It would seem that the two main rooms were always meant to be lived in as there is a payment for the 'lodging rooms'. Moreover the vast sums of money spent on furnishings and equipment implies that the Castle was used for more than just picnics. Food may have been prepared in the little stone bothy behind the castle. The accounts list French China, silk, French Chintz, and paintings. The largest sum went on furniture from the fashionable London firm of Mayhew and Ince , much of it in the 'Gothic style'. By the end of 1792 the work must have been all but complete.
The architect for Clytha Castle was thought to have been John Nash to whom a payment appears in William Jones’s account book. But it now seems that it was designed by a relatively little-known garden designer and architect called John Davenport who had a small but flourishing practice in Wales often employing the Gothick style. Jones himself is likely to have played a part in the Castle’s design, which accords perfectly with the Picturesque philosophy that was widely held amongst rich country gentlemen of the time.
At some date in the 19th century, the Castle was enlarged to make it habitable as an estate cottage. It is unlikely that the tenants occupied the two main rooms. According to one suggestion, the sitting room was for use by men and the main bedroom for the ladies. At one time three families lived here and so the empty tower must have been a separate dwelling. Mr Jones died in 1805 and left the Clytha estate to his great-nephew, another William Jones. William Jones the younger demolished old Clytha House and built the present Greek Revival house, designed by Edward Haycock and completed about 1828. In 1862 William Jones assumed - or resumed - the name of Herbert, from which family the Jones of Llanarth and Clytha descended. He died in 1885 leaving the estate to his son, Reginald. Reginald had no son but his second daughter, Gwladys Herbert, lived at Clytha until she passed the house to the Welsh Office about 1950. The estate is now owned by the National Trust, from which the Landmark Trust acquired a long lease on the Castle in 1973.
In the original building accounts for the Castle, Lodging Rooms are mentioned so it seems likely that William Jones used to stay here for short periods. It would also have provided a destination for walks and for picnic parties as well as a retreat. At some point in the 19th century, extra rooms were added to the main tower: the present kitchen and dining rooms, and the bedroom and bathroom above.
This was probably done to make the Castle habitable by an estate employee, who would also have acted as caretaker for the main rooms. From 1936 until 1947 the gamekeeper, Mr Price, and his family lived here. By the greatest good fortune his daughter, Mrs Smith, was for many years our Housekeeper. She described for us the Castle as she knew it, and what it was like to live in what must have been one of Wales's most unusual estate cottages. Even then the sitting room was kept for Miss Herbert's private use, and she often used to spend an hour or two there.