Using a hand mixer or a rubber spatula, mix cream cheese, brown sugar, peanut butter, and vanilla until thoroughly combined. Fold in most of the toffee pieces, reserving a few to sprinkle on top. Transfer to a serving dish and top with remaining toffee.