merge small clusters in R

In R, I have cut a dendrogram into clusters. However some of the clusters
have only few samples. How can I merge the small clusters with nearest big
cuter.

hc <- hclust(dist(USArrests))
plot(hc, cex = 0.6)
rect.hclust(hc, k = 4, border = 2:5)

It gives one cluster with only 2 samples. How can I merge it with nearest
cluster?

Thanks
S.