Is there a dedicated sandboxing doc

Yes. See Sandboxing. For Docker-specific setup (full gateway in Docker or sandbox images), see Docker.
Can I keep DMs personal but make groups public sandboxed with one agent
Yes - if your private traffic is DMs and your public traffic is groups.
Use agents.defaults.sandbox.mode: "non-main" so group/channel sessions (non-main keys) run in Docker, while the main DM session stays on-host. Then restrict what tools are available in sandboxed sessions via tools.sandbox.tools.
Setup walkthrough + example config: Groups: personal DMs + public groups
Key config reference: Gateway configuration