How can I sort by the id of a ManyToManyField in Django?

I've got a ManyToManyField in a user object and it's used to map the users that user is following. I'm trying to show a subset list of who they have most recently followed. Is there a trick in .order_by() that will allow me to sort by the id of the ManyToManyField? The data is there, right?

# (people the user is following)
following = models.ManyToManyField(User, related_name="following", blank=True)

theuser.following.filter(user__is_active=True).order_by("user__id")

That will give me a list of the users the user is following but ordered by when they joined. I want the order of the following list to be in order of when the user followed them.