If the shot is locked off (no camera movement) and nothing passes infront of the hole, and there's a visible clean section of matching wall in the shot, then it really is as easy as using the clone stamp tool. You could also duplicate your layer, mask off a section of solid wall, and reposition it over the hole in the first layer.
If the camera is moving you may need to use Motion Tracking or Motion Stabilisation.
If object pass in front of the hole at any time, you may need to use masks on your upper replacement layer.
I have spent two days trying to find a solution to a similar problem. I actually needed to remove some patches on a wall for several shots. For the static shots, the Clone Stamp was all I needed. But then I had a shot where the camera adjusted to account for the movement of the actors. Suddenly, the Clone Stamp alone no longer worked. I figured all I had to do was track the shot and then combine it with the Clone Stamp and problem solved. Not so much. For some reason it is more complicated than that. I have seen some overly complex solutions that probably work but trying to figure out the explanations is like trying to understand someone describe drag coefficients of rotary wing aircraft. While I am sure they work, they are way above my comprehension. I wanted an intuitive and straight forward solution for something that shouldn't require some arcane technique to fix.
What I did:
1) I used the Clone Stamp tool to paint over the spot on the wall I wanted to remove.
2) picked a more easily discernible point of reference for Tracking. In my case, I used a poster on the wall.
3) As per the instructions here, https://www.video2brain.com/en/lessons/removing-an-object-with-clone-stamp, I pasted the Attach Point data from the Tracker into Clone -> Transform - > Position
However, that didn't work for me as the object I was painting was not the object I was tracking. The solution from the video was simply moving my cloned spot to the tracked object instead of leaving it where I wanted it. So...
4) I then manually adjusted the Anchor Point of the Clone back to the original spot I wanted to cover.
I played footage and the cloned region successfully moved with the spot I was covering.
This is what I was able to figure out. I'm not an advanced user. I only use After Effects every couple of months when I need to fix minor things in footage, so I've only learned to use what techniques I need for the job at hand. No doubt, experienced users may shake their heads at my simplistic fix. I'm sure my technique would be wholly inadequate for situations far more complex than fixing spots on a gray wall, which is why my eyeballs cross when reading suggestions that seem way more involved than one would expect for something as seemingly minor as this. It's just a spot! It should be super easy to remove, right? Well, that's what I was looking for and that's what I came up with. I hope it helps.