My idea of the problem was the following:
1) I know, how many K-ships can be placed on the FREE board.
It's a rather simple formula
2) After that I check each placed ship. For him I found all the K ships that cannot be placed on the board thanks to this ship and add them to the special array (if they aren't there already). After that I subtract the number of elements in the array out of FREE board amount and that's the answer.
Who can give me test 4?

I didn't think about the case K=1. But now I fixed it and have WA 11. What's that test about?

Maybe you don't check when n<k or m<k (than common formula return incorrect value)

excuse me, can give me test 4?