When you are in what you think is a casual relationship - What does he mean when he says you would fall for me or you've fallen for me?

He is suggesting that you would or have fallen in love with him. If this is just a casual relationship you should reinforce that with him so he knows and is not expecting or thinking more. (unless of course you want more)