I did similar with a "round" world generator I was playing with, I put the world through a few passes of a check for how many points around it. Checking 8 around it would work for smoothing, but not for removing the small biomes. To do that you would need to use a wider area to check.
From my testing, the wider the area, the more small areas are cleaned up, the more times it is run, the smoother the edges. One difference with my generator is that this was done at the end of each step, as my world generator subdivided until it reached full size. ( Square-Diamond algorithm)
Here's an example of before and after:
As for getting it to work on infinite generation, just don't perform the smoothing on the outer edge, so there's always something to sample. Make the border is as wide or wider than your scan range, so you never try to access outside of what's generated (if 17x17, or 8 radius, then atleast
Of course this isn't always the fastest ever, so you have to balance accuracy and speed.