Multimedia applications, such as streaming media, will become very popular in future heterogeneous wireless networks. There are two main reasons for difficulties in the smooth transmission of a streaming media application during user movement: (1) packets may get lost due to the re-routing caused by. handoffs; (2) due to the disparity in the amount of available bandwidth among wireless cells, handoffs may cause congestion. We propose an end-to-end multipath transmission scheme that provides a comprehensive solution for the smooth handoff of streaming media. In the proposed scheme, multiple paths are acquired during the handoff period. Multipath transmission is integrated with a source adaptive multilayer encoder in order to make the streaming media more adaptive to the heterogeneous environment of networks with different bandwidths. Protection of the more important video layer through duplicated transmission on multiple paths is carefully designed for smooth handoff. The performance of the proposed multipath handoff scheme is evaluated and compared with existing schemes through extensive simulations.