Is afford gerund or infinitive?

It seems that 'to afford' is the infinitive while 'affording' is the gerund. According to grammar books, the infinitive is used after 'afford', though people (even native speakers) may use the gerund. However, what about the gerundial complex? It looks correct to say 'I cannot afford you/your attending the courses', isn't it?