this above isn't a good sign. When I ride dirtbikes all the corners have gravel..lolI stopped riding my dirtbike was that I took a spill on a corner that had a patch of gravel.
if you're scared to ride your bike in the dry you should avoid the rain. unless you want to get the experience then by all means go ahead, you have to learn sometime. go easy, be smooth, don't lean as much. just ride slower and conserve traction. riding in the rain isn't that bad, the worst of it is getting wet and dealing with reduced visibility.. with good gear it's just another fun day on the bike.
I suggest riding your dirtbike more on gravel and uneven terrain. this will teach you how to control an "out of control bike" since you'll be dealing with all the spins and slides that come with dirt riding. when bad things happen on a bike you usually don't have time to think. this is when experience and instincts take over. Riding a dirtbike will help you build these instincts faster.