It might be easier to first generate an ordered list of N unique numbers that are between A and B, and then take randomly from that (or shuffle it after it is generated).
The ordered list can then be thought of as N+1 "gaps" between each number, where each gap is >= 1 and the sum of the gaps is B-A. If you can work out what the probability of each gap size is then you could create the list recursively:
1 2 3 4 5 6
| List gaps(int n, int a, int b){ if(n == 1) return {random(b-a)}; int nextGap = randomGap(n,a,b); return {nextGap} + gaps(n-1,a+nextGap,b); } |
I don't know what the probability distribution of "randomGap" should be but I bet a statistician or wikipedia will.