In a way you are correct that there is no physical owner ship unless you have a paper wallet or some sort of hardware device upon which you can store the Ark. Interestingly, Ark does have both of those features: i.e. hardware wallet (ledger nano s) and a paper wallet. The idea of storing your passphrases is simply a security precaution much like putting the coins on a flash drive an encrypting that said flash drive. The encryption however comes from Ark itself by giving you the set of 12 mnemonic words that will ultimately decipher and allow you to use the assets. Without those words, there is nothing that anyone can do about the wallet, even if they know full well that the address belongs to you or them. The words are paramount to the security of the assets. Thus, I believe if you know how to handle your affairs and keep your assets in check then you will have no issues of holding on to the 12 words, or 24 if you created a secondary passphrase. Overall, I do not see your concern, this system has been proven efficient not only through a mathematical sense but systematically tested to be secure. But in this world anything is possible, so a healthy dose of fear and concern is always good.