The Tower cost £400 to build, and the General referred to it in his accounts as the Gardener's Lodge or General's Whim. The original intention might have been to build an ornamental cottage for a gardener, but it is hard to believe that this plan was ever realised. The pretty upper chamber was clearly meant for the Harris family's own use, and though the plain lower room was at that time self-contained, with its own door, it is too small for a dwelling. Besides which, it had a floor of knapped flint, which would not have made it very comfortable.
Whatever his practical first intentions, the General's fit of whimsy clearly overtook them, and he ended up building what was soon known as the Castle. It served the family as a summerhouse, somewhere to have tea on a June day, while enjoying the outlook over the countryside. An elderly cousin who lived with the family at Belmont noted one day in her diary that they had a syllabub there.
The Tower also served as the focal point for a new garden laid out by the General. He had bought Belmont Park in 1801, having returned from service in India the year before. There he had been Commander in Chief of the army that defeated Tipoo Sultan, ruler of Mysore, at Seringapatam, thus establishing British rule in Southern India. The prize money from this campaign was immense, amounting to £112,000, and this, with other savings, enabled the General to retire from the army, and set himself up as a country gentleman. After looking at a number of properties, he bought Belmont, with gardens, park and a farm held on lease from Oriel College, Oxford, 265 acres of land in all, for £8,960.
To begin with, the General was busy making improvements to the house and farm, but he soon turned his attention to the gardens. Here again, the practical things came first: peach and grape houses, pineapple and melon pits, and new walls, all appear in the accounts from 1805. At the same time he was drawing up plans for extending the Pleasure Grounds. A park and shrubberies already existed to the south and east of the house, but the General planned to make a new garden west of what was still a public road called Abraham Street. The line of Walnut Tree Walk is pencilled in on an estate map of about 1803, running through an orchard, past a wood called Nightingale Grove. The completed scheme is shown on another map of 1812.
This new walk was a favourite of the General's, and he spent much of his old age pottering up and down it. Already planted with walnuts, there were in fact two paths, one grass, the other smooth gravel 'after McAdam's plan'. Between them was a wide quickset hedge, in which he had cut arbours, with seats. At the end was the tower, with ivy growing on its walls to give a genuine air of Antiquity.
The 2nd and 3rd Lords Harris spent much of their time abroad on public duty, and the Tower was probably seldom visited. However, after 1870 it was given a new lease of life. When he succeeded his father in 1872, the 4th Lord Harris had already earned a name for himself as a cricketer. He made a cricket ground in the field next to the Prospect Tower and the Belmont Eleven was soon taking on other county teams. Until the 1920s, when Lord Harris was forced by age to retire from the game, cricket was a regular feature of Belmont summer life. The Tower served as a pavilion, with hooks for the gear fixed to the walls. The studs on the players' shoes pitted the floorboards of the upper room, and one distinguished visitor signed his name on the plaster.