no, DON'T change the difficulty
Your game is scoring 4.57 as of this writing, which has only ever happened for a small handful of flashes, and maybe three other games (RaidenX, Red, and an archer defense game, all of which are easy to play) on all of Newgrounds. Do not change anything. Making a game too hard will kill your score.
A few people may complain in reviews about the game being too easy, but they still give you good scores anyway. People who think a game is too hard will silently vote it down when they lose, and for something scoring as high as yours is, one 0 cancels out five or ten 5's. The easiness of the game is part of what's keeping its score up.
Good job on this.