Actually, the gap is enormous, comparible to the USA (starfleet) and Iraq (Empire).

As near as I can figure, the fastest ships in Star Wars (like the Millenium Falcon) could make '.5 past light speed'. I would guess it Trek terms this would translate into warp 1.5.

The Enterprise in TNG could get up past warp 9.6 I believe.

Plus there's the shield and weapons advantage Red already mentioned. The Imperial ships wouldn't dent the shields of the Starfleet ships, and Starfleets phasers would cut through the Imperial shields and hulls like a hot knife through butter.

And yes, Starfleet does have 1 man fighters. They were used in the DS9 episode where Sisko retakes DS9 from the Dominion.