The big challenge is the LOD of the land. I can now draw huge drawing distances as I can easily lower the LOD of the land. Once people can start digging in the land that becomes a whole lot harder and will cause quite big movements between the different LOD levels. Does anyone have experience with anything like this?
LODing is usually done per chunk. A naive implementation will base LOD on distance, but you typically want to work against a (visual) error margin for each chunk. You can easily use a mipmap approach to create LODs (for each chunk), and then calculate the cumulative height error (for each LOD of each chunk).
This is something that is a tad too heavy to run every frame, so you'll spread it out over a few, and update/recalc the error values once somebody actually modified the terrain by flagging chunks dirty.
End result would be that very rough chunks have a high (visual) error with progressive LODs, and will therefore have a higher LOD selected when rendering. (a bias, so to speak)