A User with many Addresses, but User has a primary billing, shipping, and profile Address

A User with many Addresses, but User has a primary billing, shipping, and
profile Address

I'm building a Rails app where a User can have many Addresses. A User also
needs a primary billing Address, primary shipping Address, and primary
profile Address. These primary fields can point to the same Address. A
User cannot have more than one of any primary Address.
I've created a join table called AddressInfo, and I'm bouncing between a
few options:
Make 3 columns on the User model corresponding to each of the primary
Address ids (this would remove the need for the join model I think).
Add a primary boolean column to AddressInfo, and make sure only one is
true when scoped by user_id, address_id and purpose (purpose being
billing, shipping or profile).
Add a primary date time column to AddressInfo, and use the most recently
updated as the primary address (scoped like option 2).
Maybe these options aren't the best, but it's what I've come up with so far.
Any help on how to resolve this issue would be appreciated.
UPDATE:
To be clear, once an Address is created it should always belong to that
User and be undeletable. Ex. a User changes their primary billing address
to a new Address: they should still be able to retrieve that old Address
(maybe even make it a primary address again). If I go with option 1 and
remove the join table, that means I'll need a user_id on Address.