Nightmares
The Nightmares are a race originally created for the (now cancelled) Reciprocity campaign; they are not intended to be a conjecture upon Volition's plans for extending the FreeSpace storyline and universe.
Background
The Nightmares are a newly discovered species, apparently antagonistic towards the Shivans. It is unknown whether the Nightmares are capable or willing to communicate with GTVA vessels.
Xenobiology
Much of Nightmare biology is unknown, but they appear to be a type of colony race, formed from clusters of billions of intelligent (the degree of individual intelligence is unclear), multi-purpose single-celled organisms. It is believed that all Nightmare vessels are formed from these cell colonies (in a manner analogous to coral), with cell groupings taking on specific tasks within the vessel in order to act as pseudo-subsystems.
In this sense, the Nightmare race is indistinguishable from their ships or indeed any other technology. The origin of the Nightmares, and the evolutionary conditions that drove them into space, remain completely unknown.
Development history
The Nightmares were originally conceived to act as a new enemy for the Reciprocity campaign, which was later cancelled. Their main role - despite numerous storyline changes - was to act as a antagonist and balance to the Shivans, who (in the background storyline) had destroyed Capella (specifically to seal off that area of space) as part of a war spanning hundreds of thousands of years. Essentially, the Nightmares were to act as a blind force whereby it was the consequence of their actions upon the Shivans and GTVA that would drive the storyline.
This can be seen as following the (somewhat cliched) idea of the Shivans being a symptom of a greater problem, where that problem was perpetual war between 'lesser' and 'greater' races. One of the key story elements in the Nightmare-Shivan war was to be that the Nightmares had destroyed the Shivan homeworld, driving the Shivans into space and thus necessitating their physical adaptation to zero gravity.
The general Nightmare concept shares some ideas in common with other science-fiction races, specifically the viral-like spread of the Replicators (Stargate SG-1) and the organic ships of the Shadows & Vorlons (Babylon 5), although many of these ideas were arrived at coincidentally. It is worth noting that both the Nightmare concept and story role was altered significantly over time, particularly relating to Nightmare-Shivan history.
The standard Nightmare design is a jet black, insectile ship with traces of purple and primarily green; green was adopted owing to the visual similarity between purple and red (as seen on Shivan ships). The original inspiration was primarily from the design of Shadow vessels (Babylon 5), but grew to encompass a number of visual reference points such as gothic architecture, and organic structures like coral or flower petals. Technical limitations did serve to constrict the possible Nightmare designs; the 'ideal' Nightmare ship would probably be an amorphous translucent blob, capable of morphing into different shapes and splitting into (or combining from) multiple independent ships.
Despite never having been seen in any completed campaign (although they were adopted for several ongoing or now-cancelled campaigns after the cancellation of Reciprocity), the Nightmares have attained a small degree of name-recognition amongst the community - even if only to serve as an example of over-ambition.
Usage of the Nightmares
Although Reciprocity has been cancelled, several campaigns made use of at least one Nightmare ship. Some of them are listed below:
