The short answer is yes, of course they do. Anyone venturing into the field of aviation is also volunteering for a career of regular and comprehensive testing. These evaluations take the form of oral tests, written tests, simulator check rides, and line checks in the aircraft from both company designated "check airmen" and FAA inspectors. It's a never ending cavalcade of evaluations from before an aspiring pilot ever steps foot into an airplane all the way to a few months before retirement.
And oddly enough, aviation is one of the few professions that requires such comprehensive testing. Doctors and lawyers, for instance, are mostly good to go once they finish school, though some doctors are now having to test to maintain certification every five or ten years. As a general rule, most professional pilots undergo a formal exam or "checkride" every year, but are always subject to no-notice ride-alongs by evaluators.
Testing from the Start
Each and every certification that a pilot must achieve along the way from neophyte to seasoned captain is accompanied by both written and flying examinations. I took the military route which involved a year of intensive flight instruction to get my wings followed by months of more training in the specific aircraft I was assigned to fly. During USAF undergraduate pilot training (UPT) there were checks for the first solo ride, contact or basic acro flying, instrument flying and finally formation flying in the T-37 primary trainer. It was all then done again in the T-38 supersonic advanced trainer aircraft.
Besides the designated check rides, each daily ride was graded by the instructor. No pilot was ever more than three failed rides from being booted out of the program. That tended to help student pilots keep focus. Once having graduated, check rides were given for each new certification such as air refueling, night bombing, or in the case of the Navy, carrier qualification. The military has dozens of qualifications depending on the aircraft and mission.
Back to Square One
Back in the civilian world after separating from the service, it mostly all had to be done over again. All of those military ratings count for naught in the civilian world. The FAA does throw a bone to ex-military pilots by granting them a commercial instrument rating for their military training. It's not nothing, but it's not enough to get hired by an airline. For that, a certification known as an Airline Transport Pilot, or ATP rating, is needed. This involved, you guessed it, another checkride with an FAA examiner in an airplane you had to rent.
My future employer at the time also required what is known as a "type" rating, which is a checkout to fly a particular model airplane, in this case a 737. The FAA at that time did not allow checkrides to be flown in the simulator so I had to rent an actual 737 to fly my checkride. It was only $50....a minute! To top it off, the airport we took off from, Seatac, went below minimums while we were doing touch and gos elsewhere so we had to hold for nearly an hour before getting back in. It was spendy, but I got my type rating.
Most all airline training today is done in high fidelity three axis flight simulators. These machines, which cost millions of dollars, can faithfully reproduce nearly all flight regimes with very realistic computer generated visuals. The computer can generate any kind of weather that might be encountered to include thunderstorms with lightning and can even insert wayward air traffic that must be avoided.
The simulators are so good that the first time most airline pilots fly a new model plane is on a revenue flight with passengers. The FAA certifies each simulator to make sure they reproduce the airplane's characteristics accurately. The least favorite part about flying these machines though, is the emergency procedure.
In day to day flying, almost nothing ever goes wrong. In fact statistically, an airline pilot today is very unlikely to ever experience an engine failure or other major system failure. But that doesn't mean that these things don't happen, and the simulator is how pilots are kept prepared for these very unlikely events.
Engine failures, fires, hydraulic leaks, landing gear failures, loss of pressurization and flight control malfunctions are just some of the myriad emergencies which can be summoned by the simulator operator. In the military we called it "dial a disaster", and two hours of emergency procedures in "the box" with your employment on the line can really scramble your eggs. Crashing the sim, otherwise known as the red screen of death means that the ride is over figuratively and literally.
What if You Bust?
Failing or "busting" a checkride is a rare event, but it does occasionally happen. Sometimes people are just having a bad day, and other times they come to training unprepared. Checkrides always include an oral exam on policies, procedures and systems, and a knowledge bust can mean the ride is over before it starts. Crashing the sim is rarely the cause of a failure. Poor decision making such as finding yourself in bad weather with low fuel is a more likely cause of an unsatisfactory outcome. Sometimes a diversion is called for.
As airplanes do a much better job of flying themselves due to automation, the emphasis has shifted away from strictly mechanical proficiency towards decision making and teamwork. Crews are expected to work together to solve problems. A recently adopted paradigm for most airline training does not allow for individual failure, but rather the crew will either pass or fail together, just as in the airplane.
If a crew fails a ride, they will be given some remedial training and then take the ride over again. There may or may not be some loss of income during this process as they won't be allowed to fly until passing the retake. In very rare cases, a pilot ends up having more difficulty or has a weakness uncovered. This almost never happens, but the process can lead to being let go for pilots who can't pull themselves together in the sim.
Airline passengers who have the misfortune of finding themselves on board an airliner with a mechanical difficulty should rest easy in the knowledge that the pilots flying their plane have most likely seen and successfully dealt with the problem many times before it actually happened.