I thought the term is more commonly known, but it seems it isn't. Vectorballs were a funny thing to make graphics demos. They are mentioned in several old demos and you can also google the term and will find results from the demo scene.
A little mention:http://en.wikipedia.org/wiki/Unreal_%28demo%29
Basically vectorballs are glorified particles. They have been used because they are very simple to compute and efficient to move. In the demos they were used to shape more complex objects out of points, and have funky looking transitions between the shapes.
I guess it's once again a technique of the past, and I missed the point when it was superseeded with something better.