Testing

Keywords: patience, time, circumspection, purposefulness

General hints

• Always be reasonable and logical: If you suspect that a part is buggy, then test and fix that part. If you're afraid to discover a design flaw, someone else may discover and report it in your release thread.
• Invest time and effort into the mission. The more time you spend on your mission, the better it will become, the more ideas you will get how to make it better.
• Don't be annoyed by discovering bugs you can fix. Missing head-anis, forgotten escort list priorities are common occurrences. If a debriefing shows up at a bad time every time you test your mission, open up the debriefing window, look at the SEXP tree and think slowly, logically and you'll discover the error. It's commonplace, but people learn from mistakes.
• Keep adding new things to missions if you feel like. Nothing too radical, though, as a newly added destroyer with fighter wings would turn the mission upside down. Add minor stuff, like some more messages to instruct the player what to do. If needed, sit ten, twenty or even thirty minutes looking at the grid of FRED while thinking into the possible ways of adding new items.
• Use the Designer's notes or pen and paper. Note down bugs you do not want to fix in the moment you discovered it, note down anything you will want to add. Keep erasing your notes as soon as they become obsolete. Do not let others see your mission until your notes are blank. If collaborating, this window can be used to communicate with your fellow FREDders.
• Concentrate on fixing only one bug: Do not try to fix multiple bugs unless you are sure it won't need any testing after you have fixed it (typing mistakes, no ships in Escort List, etc).
• Never release a mission without testing its last version twice or three times. A seemingly minor change like changing wing delays can result in unexpected difficulties.
• If you have the connections, give it out to others, preferably from any skill level, to test for difficulty, pacing, etc. What works for you might not work for others.

The testing process

There are diverse ways to test a mission. It's a matter of personal preference, so this article can't be all-encompassing or applicable to all types of mission. Some of the most common mistakes (or oversights) can be found here. For a more comprehensive list of possible mistakes, see this article.

The first few test runs will likely turn out to be overwhelming. You'll notice several bugs: Events or mission goals that don't work, or ships that don't behave as intended. Make your mission operable first, remedying only fatal mistakes that prevent the player from completing the mission. Mission-critical ships not jumping out belong here. Note down anything you can find, but don't want to fix all that at once. Some easy-to-fix oversights that require only a minimal effort may be fixed now. Others that require some further testing and SEXPing may be delayed a bit.

Now that you know how your mission is played, you must finalize some parts of the mission's gameplay. Decide which ships the player will have to protect or destroy and stick to it. Adding some more mission-critical craft will require far more additional testing and refinement than you may think. The sooner you make such drastic changes to your mission the less you'll have to spend with testing altogether. This doesn't mean that your mission's balance must be perfected in this stage. Minor alterations like adding one more wing of bombers or demoting the player's wingmen from Major to Captain can be made much later. What's important here is that you won't have to create a dozen new SEXPs because you've decided that the player must defend two Argos instead of one.

Test every imaginable outcome that a player might produce. Even though there aren't enough bombers to destroy a capship that you must defend, you must nevertheless take into account what would happen if the bombers did destroy your capship. You may not have a "Return to base" directive show up and there may not be a debriefing stage that deals with such an outcome. Test only "imaginable outcomes," however. In Their Finest Hour, there is really nothing the player can do to save the Colossus and the Ravana is sure to die. No thought is given to that possibility because no player can be fast and destructive enough to take down all four of the Sathanas's forward beam cannons, so it's not an "imaginable" outcome. The Colossus can only be saved using cheats. The mission designer in such cases cannot be made responsible.

In later stages of development, it helps not to test your mission, but play it. To clarify, you must not be specifically looking for mistakes like you do earlier, but try to see your mission through the eyes of a player who hasn't seen the mission yet. You may notice some inconsistencies or logical flaws between the briefing and how the mission actually plays out.

In the last testing stage, you should concentrate on minor details like adding cargo to cargo containers, prettifying your background, or adding some more mission chatter. This stage will take a lot more time than the previous ones, because you won't be hunting for something particularly obvious or observable. As a general rule, note down anything you find and after a few more test runs, once you have accumulated a long list of observations, decide which ones are worth it. You'll also have to finalize your mission's difficulty. An adequately balanced mission requires several additional test runs which are solely devoted to finding the "right" difficulty. For more information and tips for balancing your mission, see Mission balance.

Testing someone else's mission

The testing process

Remember that by the time you receive the mission file, it's supposed to be more or less complete. For the first test run, play as if you were any casual player who is playing a user-made mission for fun. Don't specifically look for mistakes yet. If you nevertheless find some, those mistakes must stand out. These mistakes will likely come up every time you play the mission, so they are to be reported and fixed as soon as possible. You'll now have a clue about the mission. After a few more casual plays, dig yourself deep into some real testing.

Now start testing a mission as if it were your own, except you can afford more test runs because you have a more-or-less complete mission. Use notes to keep track of your findings. Pay attention to details and test every possible mission outcome. You may win the mission totally by blowing up all the three hostile cruisers while keeping your corvette alive, but try to lose the corvette once and see what the debriefing will say.

Lastly, open the mission in FRED and look around. You may discover some unused Events, a forever-invalid mission objective. You'll be able to point out some mistakes that would otherwise require a dozen re-runs to be found.

It's worth noting that a mission featuring checkpoints may require several extra attempts to be adequately tested. Checkpoints were not part of the original FreeSpace games, and are a product of fans skills at generating advanced SEXPs and scripts. From a FREDding standpoint, the implementation of checkpoints add a considerable degree of extra complexity to the structure of any mission, and said complexity requires testing. A single, triggered event that does not take into consideration the checkpoint infrastructure may break the entire mission, so the mission itself requires all checkpoints to be tested.

Submitting feedback

Submitting feedback to the mission designer is the last step in your testing of someone else's mission. Whatever way of information exchange you choose (email, forum post, instant messaging) remember to be objective. Don't use attributives like "bad" or "amateurish," even if the mission designer made a very blatant mistake. Even experienced mission designers tend to forget about putting mission-critical craft on the escort list. A factual "There is no Escort List." sentence will do it. If the mission designer is a beginner, explain why the Escort List is needed. What's obvious to veteran FREDders may not be obvious to beginners. You may also wish to avoid curse words and other forms of unpolite phrasing in your testing report.

Don't report all your findings. A disproportionately long list of mistakes is discouraging. Concentrate on the really important mistakes first and put forward some propositions on how to fix them.

While it has previously been said to be as objective as possible, some praise—which is inherently subjective—helps maintain a steady level of encouragement. Be specific about your compliments, also. A very general, "The mission was excellent, but I have a long list of mistakes here" sentence sounds dishonest. A dishonest praise is no praise. It is okay to praise the mission for its intensity and point out some minor briefing icon-related mistakes.

Differentiate between a "mistake" and a "designer choice." As a tester, you may notice that the briefing uses no colored text, but instead of portraying this absence as a mistake, ask the creator whether it was his intent to use one color only.

Types of findings

Testing is a rather complex process where several factors tend to converge: the process normally results into the finding of objective bugs, recommendations on how to improve the mission, and what may be perceived as plotholes or inconsistencies. A tester has the right to provide feedback on all such aspects, but testing reports should be differentiated for a number of reasons.

For example, if a critical ship is not guardianed in FRED and is also meant to survive, its destruction will result in a series of oddities, such as its messages being sent by Command, general mission unbalance, and a major plothole should the ship be featured in any of the following missions and/or be mentioned in debriefings as an operational asset. The FREDder may have simply forgot to guardian said ship without noticing it (e.g., the warship survived during his playthroughs but didn't during yours). This is a mission bug and should be mentioned as such.

The above mentioned warship engages another warship during the mission via an AI attack order, so its maneuvers may follow unpredictable patterns and the armaments this ship is meant to fire at the opposing unit may not be used during the confrontation due to POV variations. At this point, the tester may recommend implementing waypoints and capship speed limitations to have a proper control on how the battle is likely to develop; guardianing key turrets and engines on both vessels elso ensures that the battle develops a certain way. Should a tester decide to recommend any of this, feedback of this kind is to be listed as a recommendation: the tester believes a number of changes may improve the mission, but the mission still works in its status quo and the FREDder may decide not to change what he's done.

Now imagine the following scenario: the previous mission stated that this warship is not equipped with a certain weapon, yet this magically appears and the warships starts using it with no explanation whatsoever. Or, the ship sustained a lot of damage during the previous mission, yet its hull integrity is now set to 100%. Or, Command deploys this warship, saying that it's the largest asset available, yet we know for sure that there are other - and larger - warships available. Or, any of the above things combined, if not all of them. These all qualify as plot inconsistencies, and may compromise a player's impression on a campaign the negative way. A tester has all the right to point them out, but before doing so it may be a good idea to get acquainted with the new continuity created by the FREDder, as these occurrences may indeed have explanations but you didn't quite notice them. Once you are sure these could indeed be inconsistencies in the plot, submit them to the FREDder, but do so separately from the actual bugs and recommendation lists to avoid confusion and allow the FREDder to better manage mission bugfixing. The FREDder will then realize if they're actual inconsistencies or things that are covered by the plot, but not in a way that makes them obvious to the player (in this case, the FREDder may decide to add the much needed extra information required). Some of these potential inconsistencies may be in place for the sake of mission balance (e.g., the warship now has 100% hull integrity, it ended the previous mission with 17%, and anything below 40% in the current mission may make it too easy/difficult), so a proper compromise could be setting, for example, a hull integrity of around 80%, a few damaged subsystems and turrets here and there, and an extra bit of lore explaining how the warship was subject to rapid field repair.

Speeding up your testing

FREDders, especially when making a campaign, are faced with time constraints due to real life reasons. This section will help you minimize your testing time without sacrificing effectiveness.

• Most importantly, don't quit every time after every test run. Minimizing your FS window no longer crashes FreeSpace Open. Use ALT-Tab to switch between FRED and the game.
• Changes to the mission file can be applied while the game is using that mission. Apply changes, save the mission, switch to the game, press escape and click "Restart mission." Smaller, immediately fixable mistakes (typos) can be fixed over the course of one run-through in great quantities.
• Use 1024x768 for best loading time. Also disable the mediaVPs if your mod doesn't rely on them.
• Rename your mission to make it the first on the list in your Mission Simulator. Renaming your mission to a_<mission name>.fs2 or 1_<mission name>.fs2 will be sufficient. Don't forget to rename it again to its finalized name when you start thoroughly testing another mission.
• There are no moral standards for testing. Feel free to use cheats to easily and quickly reach the part of the mission you want to test. Use of "developer events" like ones that make friendly mission-critical craft or the player craft invulnerable if you want to test whether the end part of the mission works as intended. Don't delete such events when you are done, because that would change the order of events, which is known to break the mission. Use false instead to disable them.
• If your mod makes use of a custom weapons.tbl, you can add a very powerful weapon that you use solely for testing runs. Having such a powerful laser will help you finish the mission far more quickly than with weapons that will be otherwise available to your players—this is especially useful for testing debriefing stages. To make sure that your players won't use this weapon, just don't set it as an initial weapon in the campaign file. (Use "normal" weapons otherwise for testing mission balance). If you don't want this weapon to be available at all, remove it from the tables (and the mission's weapon loadout tab) entirely.
• If it's not about testing the mission's difficulty, simply lower the difficulty level. It's less stressful and allows you to concentrate on what you want to test. When testing message delays, difficulty does not matter.
• You can set up a one-mission campaign to easily access your mission by clicking on the Ready Room, instead of using the Tech Room method. This method may compromise your pilot file when not done right, so be careful. Less experienced mission designers may not want to follow this method at all.
• Decide which changes actually need testing. You may want to test if your briefing icons are set up right, but you may not have to test spelling corrections. The more experienced you are, the better you'll be able to judge which changes require testing.
• To test if certain events work or not, use directives. Therefore, you will be able to tell when and if the problematic event turns true or false.

Testing your campaign

By the time you had set up your campaign file, you'll ideally have a series or more-or-less finalized missions. This section will give you advice on how to test if your campaign file is set up properly. Remember that changes applied to your campaign file will take effect only when you reset your campaign in the Campaign Room.

• First and foremost, your Error Checker must not indicate a problem. If it does, it's very likely something won't work right.
• Enter the Campaign Room, select your campaign, and read your campaign's description at least twice to discover if you have done some typos or made some spelling mistakes. If you discover something you want to change, don't do that yet!
• Run the first mission and discover if the player has access to all the intended ships and weapons. First-time campaign makers tend to forget that they must allow the player to use certain ships and weapons in the campaign file. If the player has weapons and flies the intended fighter by default, it's okay. If you have found anything you want to change while you were in the Campaign Room window, do that change now. If the player doesn't have access to all the intended weapons or ships and you found a spelling mistake in your campaign's description, you can change all that in one run.
• To test if your Repeat Mission SEXPs work, jump out at the beginning of each mission and see if the game tells you to replay the mission in the debriefing room. Ideally, it will. If not, revise.
• Test what happens if the player is minimalistic. Complete only those objectives that will allow the player to progress in the campaign. You can be a maximalist for the second or third re-run.
• If you complete your campaign, see what happens if you click on the Ready Room. If it says you have completed the campaign, you're done.
• If you're using custom tables and/or mods, set up a mod folder that contains only the files you're about to release. This will objectively reflect what your players will have when they download your campaign. If there's a mod you've forgotten to include in your download, this method will bring that to light.
• You needn't test your campaign as a whole in the initial stages of creating the campaign file. You can test it in small chunks of three or four missions to see if the campaign file is set up correctly. Some missions might require a complex set of conditions to allow the player to progress; this method helps a bit in testing these kind of missions.

See also

• General hints
• Mission balance
• Typical FREDding mistakes