In the near future, wide variety of wireless networks will be merged into the Internet and allow users to continue their application with higher degree of mobility. In such environment, multimedia applications, which require smooth rate transmission, will become more popular. There are two main reasons that cause difficulties in the smooth transmission of stream media application when a user roams around the wireless mobile networks: 1) packets may get lost due to the re-routing caused by handoffs; 2) due to the disparity in the amount of available bandwidth among wireless cells, handoffs may cause congestion. We propose an end-to-end multi-path transmission scheme that provides a comprehensive solution for the smooth handoffs of stream media. In the proposed scheme, multiple paths are acquired during the handoff period to reach a single mobile node. Multi-layer encoding technique is applied to make the stream media more adaptive to the heterogeneous network environment with different bandwidths. Protection of more important video layer through duplicated transmission is carefully designed for smooth handoff using multiple paths. The performance of the proposed multi-path handoff scheme is evaluated and compared with existing schemes through extensive simulations. The simulation results show that the proposed scheme improves the throughput and quality for stream media application during the handoff. Cost of the proposed scheme is also carefully evaluated in terms of transmission efficiency.