Depends on what you mean by game programming. Games have several different components, like graphics, AI, IO operations and networking. They don't all necessarily have to be built in the same language.
If you get good at Java, you'll be able to develop for a number of mobile platforms, like Android. Most of the elements in those type of games are pre-built; your programs just deal with piecing them together to create something unique.
If you are talking about something graphics-intensive that you are creating yourself, you'll need to know about low-level languages (assembly) that operate at the GPU of a graphics card. Your math skills should also be good at least up to the level of vector calculus.