The OOM killer kicks in when you run out of virtual memory, not physical memory. If you genuinely have processes that only periodically actually need their heap to be large, but don't return unused memory to the OS, you can simply allow the OS to page out the address space that isn't currently used. There are subtle differences between returning address space to the OS, and simply not using address space, but they aren't the kind of differences that impact your problem.

G1's heap sizing logic is readily adjustable. The old defaults did rarely return memory to the OS, but you could tune them to suit your needs. Either way, this is no longer accurate an accurate representation of G1's behaviour as the runtime has adapted to changing execution contexts: https://bugs.openjdk.java.net/browse/JDK-8204089