Television networks are in the midst of their annual process of testing new shows to decide which ones they’ll pick up. The typical test audience is made up of 50 people, recruited based on age, sex and race, and a network’s core demographic. The audience members watch the show, judging it via a handheld device that has a knob enabling them to express like and dislike. Not every media outlet embraces the concept of test audiences, and the testing services are modifying their criteria to better reflect today’s viewing behavior.

The Wall Street Journal reports that Amazon is one such media outlet that has reinvented audience testing. It posts pilots of new shows on its website and allows all comers to rate the shows before it moves forward. Among new media outlets, two that Comcast invested in, BuzzFeed and Vox, also “see the potential to use online videos as a testing ground to see what might develop into TV shows.”

Testing company Screening Engine/ASI now tests whether or not shows are “bingeable,” in one instance showing test audiences 10 episodes of a show over four days, followed by “intense focus groups,” to determine “at what point during the season” the audience buys into the show.

“It’s like putting on a wedding each time you do one,” said chief executive Kevin Goetz, about the testing process.

TV executives are likely to agree with former NBC Entertainment president Warren Littlefield’s assessment that testing is “absolutely gut-wrenching.” Littlefield is now an executive producer of FX’s “Fargo,” a hit. But, he notes, many shows don’t test well, pointing out the famous example of how “Seinfeld” tested badly when test audiences didn’t find the characters likeable enough.

Littlefield observes that in today’s market, mass market appeal is less important, since so many edgy shows find “small but loyal audiences” on cable TV. “The research was to sand down the edges that possibly offended or pushed away a larger audience,” he said. “Today we seek those edges and embrace them.”

Some newer media outlets reject testing altogether, including Netflix and HBO. “Our bet is on the creative vision, and you can never measure that by just looking at a pilot without any context,” said HBO president of programming Mike Lombardo. But others still rely on it. Showtime chief executive David Nevins reports he uses testing to drill down into how audiences react to specific stories and characters, and see if they are “confused, into it or bored.” That helps with the editing process, he notes.