This can be a very tricky process indeed!! Even I can get it wrong especially when a bunny is very small or young. I must admit it is much easier when your rabbit is older as there are obvious signs which i will explain.
An older male which is laid on it's back, you can usually see their testicles, more so on a hot day, this enables them to stay cool. They are what i describe as two kidney shaped items found near the genital opening near the base of their tail.
If your rabbit is young/small then you may need to seek out the genitals. To do this place a finger on each side of the genital area and place a small amount of pressure and see what shows up. A male tends to have a tube pop out with a round opening at the end.
this is an older male rabbit.
this is the tube i mentioned earlier pictured above.
Females also have genitals that 'pop' out too this is how mistakes are easily made.
Once again you go through the same routine as mentioned above, hold your bunny on it's back, place 2 fingers around the genital area and put slight pressure on that area and again examine what shows. A female tends to show a raised area with a high point which has a slit that tends to be sloped to the anus area (bottom).
Hope this helps, and the more you do it the easier it gets!