Ferrari -
Many students have a difficult time at first, keeping their right (picking) hand fingers close to the strings. So don't feel like you are alone regarding this issue.
I assume you are anchoring at least your pinky on the banjo head when you are playing. You may also attempt to anchor the ring finger as well.
You do not mention how long you have been playing. Too many banjo students are concerned about speed. Try to be patient and the speed will develop over time. I suggest you start every practice session with rolls and play them slowly. Gradually build your speed. Focus on keeping your fingers close to the strings, while anchoring your hand with the pinky and or ring finger. Practice rolls daily and play them repetitively. After awhile, stop looking at your right (picking hand) while you are playing the rolls. Try this (playing rolls) at least 10-15 minutes per day.
Over time you will notice improvement. As you mentioned, do not give up. Working hard pays dividends. Good luck.
Dave