Try keeping the JPanel, and adding the Canvas3D to the JPanel instead of replacing it. Then the Canvas3D becomes a managed component and shouldn't go outside the application bounds, and should resize properly, etc.
Also, the Canvas3D is a heavyweight component, so it will show up over any lightweight Swing components, including menubars, toolbars, etc. if they are in the same container. So ideally your Canvas3D should be the only component in the JPanel. Your other Swing components should be in a different JPanel. Then you can put both of the JPanels into another JPanel to organize them however you want, using LayoutManagers.
If you have popup menus and you want them to show up over the Canvas3D, then you can use:
1
| JPopupMenu.setDefaultLightWeightPopupEnabled(false); |