The walls are patrolled by Winged Warrior Sentinels. If players attempt to climb up the wall but make too much noise, they will alert the Winged Warrior Sentinels who will pour down stones doing 1d6 damage to the players climbing the wall and require them to make a DC 15 Dexterity Save or fall off the wall. As the Dungeon Master, you can determine the height they have fallen from and if they are lucky enough to grab a handhold as they fall (I would make it the closer they are to the Dex Save).
If the party manages to get to the top quickly, they can hide to gain advantage on the passing patrol on top of the wall. The patrol uses crossbows. If the party does not go the wall route and attempts to force the gate or talk their way in, they will deal with Winged Warrior Sentinels with spears instead.
Winged Warrior Sentinel
Armor Class: 15 (natural armor)
Hit Points: 20 (4d8+2)
Speed: 30 ft
Fly: 30 ft (hover)
STR: 15 (+2) DEX: 10 (+0) CON: 14 (+2) INT: 10 (+0) WIS: 10 (+0) CHA: 6 (-2)
Skills: Perception +2
Senses: Passive Perception 12
Challenge Rating: 1/2 (100 XP)
Stoneform: Winged Warrior Sentinels can once per encounter take a full round action to transform into a stone statue for 5 rounds. During this phase, their armor class becomes 20 and their hit points are restored by (2d8). This can be interrupted by a melee attack.
Spear: +2 to hit, reach 5ft., one target. Hit: 6 (1d6+2) slashing damage.
Crossbow: +0 to hit, range 80/320, one target. Hit 5 (1d8) piercing damage.