Relationship of daughter of a nephew?
Well the daughter will be your grandaughter and you will be a grandmother/grandfathther.
If your brother-in-law is the husband of your sister, presumably his daughter is your sister's daughter, too. Then his daughter and your nephew are either brother and sister or first cousins. If the daughter is not your sister's daughter, but the child of a previous marriage, then she is could be the half-sister of your nephew or not related to your nephew at all. If your brother-in-law is the brother of your spouse, then his…
As far as I can make out, Your daughter is your fathers (common ancestor) grandchild, and your grand nephew (not great nephew) is your fathers great grandson. According to the chart at http://freepages.genealogy.rootsweb.ancestry.com/~gentutor/chart.html this would make your grand nephew, your daughters First Cousin once removed