It's only a yellow line. How hard can it be?

January 6, 2009 9:50 PM

The computer generated first-down line in American football is something we take for granted these days. However, the logistics required to make this work is pretty complex. At the very least, have you considered this: if it's computer generated on a moving image, how do they draw it under the people running around on the field, and not over them? And it gets a bit more complicated than this. "Here are some of the problems that have to be solved in order for this system to work:

* The system has to know the orientation of the field with respect to the camera so that it can paint the first down line with the correct perspective from that camera's point of view.
* The system has to know, in that same perspective framework, exactly where every yard line is.
* Given that the cameraperson can move the camera, the system has to be able to sense the camera's movement (tilt, pan, zoom, focus) and understand the perspective change resulting from the movement.
* Given that the camera can pan while viewing the field, the system has to be able to recalculate the perspective at a rate of 30 frames per second as the camera moves.
* A football field is not flat -- it crests very gently in the middle to help rainwater run off. So the line calculated by the system has to appropriately follow the curve of the field.
* A football game is shot by multiple cameras at different places in the stadium, so the system has to do all of this work for several cameras.
* The system has to be able to sense when players, referees or the ball cross over the first down line so it does not paint the line on top of them.
* The system has to be aware of superimposed graphics that the network might overlay on the scene."

Here's a detailed explanation regarding how it's done. And here's a cool video showing it in action.

posted by SpacemanStix (52 comments total)

TYFPT
posted by autodidact at 10:11 PM on January 6, 2009

I recently had a conversation about this with my non-sports watching cousin. He was terribly interested in understanding the logistics of the first down line and terribly uninterested in what everyone else was watching. This will be a fantastic read for him. Thanks!
posted by bemoedee at 10:43 PM on January 6, 2009

Don't they do much more interesting things with Live Football (Soccer) these days (since the game doesn't break as much as Football (American Padded Style) does they have to pipe the ads in around the field by superimposing them on the barriers, the players, balls and so on go on in front of them all through the game and let's face it, soccer moves faster than football and baseball... impressive stuff.
posted by NiteMayr at 10:47 PM on January 6, 2009

Very cool. Does the color palette tactic work on snowy days when one team is wearing white? Actually, the computer always has to paint over the white lines on the field. Are white uniforms always a slightly different shade?
posted by painquale at 10:52 PM on January 6, 2009

I can't speak to footie, nitemayr, but many of the "stadium" ads at US football and baseball games are CGI added to the broadcast, including most (all?) of those "behind the batter" rolling billboards.

But those seem like annoyances. That first-down line in football has always impressed me. It's like green-screening live video.
posted by rokusan at 10:58 PM on January 6, 2009

neat article

"The color palettes are also critical to the system. The computers must be able to distinguish between grass, on which the line should be painted, and everything else (players, referees, the ball, etc.), on which it should not. Color palettes solve this problem."

So... do the Packers ever get yellow lines drawn on them?
Actually, is the choice of bright yellow as a secondary colour some sort of preemptive strike at this?

It's cool this works as well as it does -
giving the variations in colours of the field itself (in different conditions - mud to snow) and ever changing lighting/shadows, that's got to be one robust palette to discriminate "field" from "not field"
posted by sloe at 11:05 PM on January 6, 2009

This system actually sounds fairly naive and cumbersome, which might be because it dates back to the nineties. The mention of two technicians dedicated to operating the system and adjusting palettes during use and so on is quite revealing in that respect, it's probably not particularly robust on its own at all.

The description of the hardware also suggests that it's old. Several computers just to process angle data from the camera heads, SGI boxes for the actual compositing... It sounds exactly like what a system like that would look like if it was designed in the nineties and unchanged since. Several huge racks of boxes to do something that's not terribly complicated with more modern technology.

The explanation of the path the data takes is kind of revealing too. The data-over-audio hackery isn't so surprising, given that it utilizes an existing data path from camera to control truck, but the encoding of it into the video signal s is reminiscent of VITC or Teletext, and is kind of a weird hack, it's probably done to make the 3D data switch along with the video signal, but it seems pretty unelegant.

I'm surprised it works as well as it does, though. The results don't look too bad at all.
posted by Joakim Ziegler at 11:44 PM on January 6, 2009 [6 favorites]

>but many of the "stadium" ads at US football and baseball games are CGI added to the broadcast

I seem to remember there was a curfuffel here a few years ago because the ads actually at Lang Park(?) were being (to be?) digitally replaced with different ads in the TV broadcast. I can't remember now if it was the venue upset with the broadcaster because without tv coverage their bill boards weren't selling or if it was the ad buyers upset with the venue because they weren't getting the exposure they thought they were buying I wish I could find some details on it now but my google-fu is failing me. :/
posted by adamt at 12:08 AM on January 7, 2009

I have never seen this done before and it is amazing.
posted by Jairus at 12:33 AM on January 7, 2009

Joakim, you're most likely correct when you mention that current technology could do the job with less overhead. But it will probably be a while until any such changes are implemented for one simple reason: The current system works. And works well. Sometimes, in technology, that's good enough, and enough to keep newer equipment at bay. (Heck, we send some relatively ancient stuff into space these days. The reason old stuff gets sent into space? Because we know it works and we can count on it to work for a long time. ) You do wonder what the next generation of first down line systems will be able to do. For example, it might indicate when the ball has crossed the line or even be tied into the scoreboard system. It'll be fun to see.
posted by azpenguin at 12:44 AM on January 7, 2009 [1 favorite]

While the "it works" argument makes sense, it isn't like the NFL and the networks who broadcast it are gun shy when it comes to trying new TV toys. Some of you may remember a Superbowl from about 5-7 years ago (on CBS) where they did Matrix-style "bullet time" shots, but of the live game (ah, here it is). And both the prime time games on NBC and ESPN use the newish Skycam shots.
posted by owillis at 2:31 AM on January 7, 2009

Unless they put some kind of beacon in the football, I don't think we'll be seeing it any time soon. Hockey had a similar concept, but it was abandoned.
posted by pwnguin at 2:33 AM on January 7, 2009

The FoxTrax puck is the only time I've ever watched hockey. Otherwise its harder to see on TV than a golf ball is after Tiger's whacked it.
posted by owillis at 2:37 AM on January 7, 2009

I actually just found out about the existence of the computer-generated line about two days ago, when a coworker called up some highlights videos during a lull.

I guess I did kind of take it for granted (once I'd gotten clarification on what exactly a first down was). I'm impressed that it's actually so very complicated to implement.
posted by Scattercat at 3:15 AM on January 7, 2009

When it first started being implemented, I remember a couple of buddies pranked another and convinced him that it was a tiny robot that rushed across the field and laid down a quick line of paint. He believed it for a few minutes before he realized that the field wasn't covered by dozens of yellow lines as the game went on.
posted by explosion at 3:46 AM on January 7, 2009

Given that the cameraperson can move the camera, the system has to be able to sense the camera's movement (tilt, pan, zoom, focus) and understand the perspective change resulting from the movement.

What. If I were building this system, I would completely any camera sensing. Just look at the image. Find the markers. Draw the line. Don't draw the line on a model and then try to tilt, pan and zoom it to match what the camera is doing.

Also, a 3D model seems like overkill not to mention a maintenance nightmare (unless all fields are identical). Just read that value from the lines you can already see crossing the entire field.
posted by DU at 5:01 AM on January 7, 2009 [2 favorites]

My only issue with this in NFL games is that I always forget that the running back or receiver on the field can't actually see the line himself. I find myself wanting to yell, "why are you stopping there, can't you see the damn line two feet past you?"
posted by octothorpe at 5:19 AM on January 7, 2009 [7 favorites]

owillis - the FoxTrax puck was a crime against humanity hockey. Fox didn't understand that good announcers are better than dancing animated robots and glowing crap on the ice. Fox also didn't understand that catching a puck during a game is a rare moment to grab a souvenir, and due to the tech inside the puck spent a good part of their coverage taking back said souvenirs from fans that paid money to attend so that the casual viewers at home could see glowing crap on the ice. Fox was trying as hard as possible to turn professional hockey into a live video game. It was a bad idea and I'm glad it didn't last.

Thank god for most of that debacle I was able to watch games on CBC. I mean, I understand what they were trying to do, but in hockey a lot of what happens depends on what is going on away from the puck. When everything was focused on the puck - and the puck glowed like the sun - it took away a lot, and was damn annoying for many fans, including myself, and I was at that time just getting into the game but still had no real problem following the puck even on my crappy TV.

sloe writes "So... do the Packers ever get yellow lines drawn on them?"

My favorite college team wears green, and there have been several times I've seen the first down line superimposed over a player's legs because of similarity to the field color. So, yeah, it happens, but I'd guess it happens less often in the pros.
posted by caution live frogs at 5:19 AM on January 7, 2009

I wondered how they did this. I assumed there had to me some kind of cleverness that allowed for a simpler method - because who would go to this much trouble?
posted by Joe Beese at 5:24 AM on January 7, 2009

Well if I were building the system, I would just build a box with a port on one end marked "in" and one on the other marked "out". And it would work by MAGIC!
posted by kcds at 5:26 AM on January 7, 2009 [3 favorites]

Yow... when I saw this (on a game) last year I was very impressed, but I figured there was just a broadcast pod and a reciever pod on each side of the field and it was being moved by PA's up and down along with the real football market. Sorry I don't know the terms, I don't do sports too well. Anyway.. suprising to learn there's all this complicated digitizing/rendering going on in real time, but hey, I guess that way was easier for them initially!
posted by cavalier at 5:38 AM on January 7, 2009

The amazing thing now is that they've gotten it working in the wire-camera that floats over the players. Doing it from fixed camera positions is child's play compared to that.
posted by smackfu at 6:08 AM on January 7, 2009 [2 favorites]

The description of the hardware also suggests that it's old. Several computers just to process angle data from the camera heads, SGI boxes for the actual compositing... It sounds exactly like what a system like that would look like if it was designed in the nineties and unchanged since.

I wouldn't read too much into it, since those HowStuffWorks articles are from 2003 which is pretty old for tech. Like I'm sure it all had to be updated to HD at some point.
posted by smackfu at 6:15 AM on January 7, 2009

I may go off-topic here. I'm impressed by the technology that made this thing work, and the first game or two I watched with the computer-generated first-down line, I sat in awe of this marvel. Now that I've had time to think about it though, I think it's made me a more lazy football-watcher. I just sit & stare at that imaginary line, waiting for someone to move the ball across it, and find I watch much less of the action on the field as a result. Flags are thrown, and I'm like "Huh? Where? I was watching the first down marker." I kind of wish there was a switch to opt out.

But then, I'm an old-fart, traditionalist good-old-days technophobe that way, when it comes to sports. I still get grumpy waiting for the guys in the booth to review the play on tape to uphold or strike down the on-the-field calls.

In other words, get your computer stripe offa my lawn!
posted by Devils Rancher at 6:30 AM on January 7, 2009 [2 favorites]

superimposing them on the barriers, the players, balls and so on

I watch a decent amount of soccer (football) and hadn't noticed them superimposing ads on the players or the balls; is this actually true?

The amazing thing now is that they've gotten it working in the wire-camera that floats over the players. Doing it from fixed camera positions is child's play compared to that.

Those cameras are awesome. There's an FPP somewhere around here about them.
posted by inigo2 at 6:32 AM on January 7, 2009

So... do the Packers ever get yellow lines drawn on them?
Actually, is the choice of bright yellow as a secondary colour some sort of preemptive strike at this?

Yup, sometimes they do... I'm coming up empty on YouTube footage of it happening, but I've definitely seen legs with first-down markers drawn across them. It's usually momentary and not a solid line--you'll see an area of the screen with grainy, flickering yellow superimposed over the green parts of a uniform that lie in the path of the yellow line, and it usually goes away as soon as there's a solid outline back in that part of the frame (I'm guessing that's a secondary error-correction measure they've built into the software: shapeless green blobs get yellow drawn over them, but green blobs with solid outlines are more likely to be players than grass, so they don't).

Thanks for this post, by the way... I love looking at the inner working of this stuff, and I'm usually the only guy watching the TV who's more fascinated by the thought of what would happen to a team with blue jerseys playing at Boise State (or the Cowboys playing on a snow-covered field) than by the game itself. The mechanisms behind this are WAAAY more complex than I would have imagined, because of what DU said: 3D modeling of the field is a classic case of overthinking a plate of beans. But, in the mid-90s, image manipulation of real-time video feed as complicated as what we're talking about here would have been damn near impossible, and this is a more front-loaded way of doing it.
posted by Mayor West at 6:44 AM on January 7, 2009

The television innovations that the US broadcasters have come up with to make boring, seemingly never-ending sports like NASCAR and American Football tolerable/understandable are truly stunning. The labelling system that NASCAR uses for cars driving around the track is brilliant, too, and pushing the limits of what can be displayed as the action is occurring is great. They were the first with kerb mounted and pit crew helmet cameras, too, which are a great way for people to see what actually happens.

I, too, have watched about 10-15 minutes of american football just to try and work out which lines were computer drawn and which were real, and what they exactly meant. Interesting as hell, when you first see it. Really impressive that they are able to do it so effectively that it can almost like like the two fellas with the flags are holding down a big stripe when you first see it.

Of course, once I got over that, the game was more boring than waiting to need a haircut. It says something about a 'sport' when you have to throw technology and flashy graphics at it to such an extent to try and not only allow the audience to actually understand what is going on, but also keep them interested.
posted by Brockles at 7:08 AM on January 7, 2009

In pro golf they've started to do something similar to the glowing puck thing. At the ryder cup this past summer they would do this thing where after a guy teed off, they would replay his drive and somehow they got the ball to leave a bright blue path wherever it went. Anyone have any video of that?
posted by tylerfulltilt at 7:15 AM on January 7, 2009

Brockles: both sports you mention were popular and interesting before the new technology.
posted by rocket88 at 7:17 AM on January 7, 2009 [1 favorite]

It says something about a 'sport' when you have to throw technology and flashy graphics at it to such an extent to try and not only allow the audience to actually understand what is going on, but also keep them interested.

Heh. I might put opera subtitles in the same category.
posted by smackfu at 7:20 AM on January 7, 2009 [2 favorites]

Actually, the virtual first down line was pioneered by a company known as Princeton Video Image (PVI), which extended the technology into virtual ad insertion and other areas. I suppose SportVision acquired it somewhere along the way. PVI filed for Ch. 11 in 2003.
posted by VicNebulous at 7:24 AM on January 7, 2009

I love the yellow line, and I don't mind the subtler but less necessary line they sometimes do for the line of scrimmage, but I hate the giant down-and-distance graphics they started superimposing recently (this year?).

The overhead wire camera (Skycam) was introduced by the XFL.
posted by kirkaracha at 7:26 AM on January 7, 2009

Of course, once I got over that, the game was more boring than waiting to need a haircut.

Could we have just one fucking sports thread where we don't do this?
posted by dirigibleman at 7:30 AM on January 7, 2009 [9 favorites]

I take that back. Seems there were two technologies used for the lines. SportVision now remains.
posted by VicNebulous at 7:35 AM on January 7, 2009

Why didn't they just use a Commodore Amiga?
posted by LordSludge at 7:36 AM on January 7, 2009 [1 favorite]

Wow... I honestly had never given this a moment's thought, but it's really cool to see the behind-the-scenes of it.
posted by marginaliana at 8:02 AM on January 7, 2009

Not that I understand the miracle of first down lines, I wish I had the technology to disappear that goddam dancing robot on fox.
posted by cairnish at 8:08 AM on January 7, 2009 [1 favorite]

I really dislike football and I never watch it, so a few years ago when I turned on the Superbowl to see if there were any good ads, I was shocked at the crazy CG shit going on on the field. I called my wife (and fellow geek) into the room to see this awesome use of video technology in a live broadcast and she just gave me this withering look which suggested that no matter how neat the effect was, I should turn the puppybowl back on. Right. Now.

Did I mention that she hates football even worse than I do? Also, she loves the dogs.

Sadly the puppybowl did not make use of a virtual first down line.
posted by quin at 8:16 AM on January 7, 2009

For everyone who is saying how outdated the tech looks, I would bet the farm that the article and video are both showing some pretty old-school yellow line tech. As an NFL fan (OH NOOOOOOO) I can say with certainty that the yellow line has improved markedly in the past 3-4 years. It used to be wrong, superimposed on players, spotty and generally low-tech pretty often, and now it is almost 100% accurate. There is no way they are still using this technology.
posted by jckll at 8:59 AM on January 7, 2009

I really dislike football and I never watch it

I'm not much of a football fan either, and don't often imagine myself getting excited enough to make FPP's about it (which I just did here). But I'm forever fascinated by cool CG stuff like this. I'm really impressed when it's made to look much easier than it is, almost to the point that you don't notice it anymore.

And yeah, my wife could care less about CG stuff too no matter how super awesome it is. But I guess that's more understandable than me missing some sort of a sport-appreciation-gene, so can live with her humoring me occassionally when I get excited about it.
posted by SpacemanStix at 9:12 AM on January 7, 2009

This is a nice post. Thank you.
posted by wabashbdw at 9:16 AM on January 7, 2009

No matter how high tech the yellow line, verifying that the ball crossed it is still as low tech as it gets.

I'm waiting for the day when turf is made up of -- or includes -- fiber optic "grass" that can display whatever needs displaying.
posted by schoolgirl report at 10:26 AM on January 7, 2009 [1 favorite]

Fox didn't understand that good announcers are better than dancing animated robots and glowing crap on the ice.
You do understand that poker... freaking poker... does better ratings on TV than hockey, right? So perhaps hockey fans shouldn't have been so quick to turn up their nose at the glow puck (and other tv gimmicks/innovations) and then hockey wouldn't be so low on the totem pole here.
posted by owillis at 10:51 AM on January 7, 2009

I wish I had the technology to disappear that goddam dancing robot on fox.

But it was so cute when he got Guitar Hero for Christmas!
/barf
posted by kirkaracha at 11:16 AM on January 7, 2009

The FoxTrax puck is the only time I've ever watched hockey. Otherwise its harder to see on TV than a golf ball is after Tiger's whacked it.

I got into a hockey-watching phase rather suddenly around 1994. At first, it was difficult for me to follow the puck, and had FoxTrax existed at that time, it might have been helpful to me—for about the first 2.5 games I watched. After that, I was able to follow where the puck was by the actions of the players, even when the puck was not visible on screen, and FoxTrax would have been just annoying (as it was to me when it was introduced). It's really not that hard for someone even slightly familiar with the game.
posted by DevilsAdvocate at 11:28 AM on January 7, 2009

My only issue with this in NFL games is that I always forget that the running back or receiver on the field can't actually see the line himself. I find myself wanting to yell, "why are you stopping there, can't you see the damn line two feet past you?"
posted by octothorpe

Octothorpe's favorite football team sucks.

or at least needs a better running back... dude, you gotta know that shit!
posted by butterstick at 12:08 PM on January 7, 2009

I should turn the puppybowl back on. Right. Now.

I have to say, whenever I feel like I hate my job even the tiniest bit, I reflect that I could be the poor son of a bitch who has to throw a penalty flag and then wipe dog's asses on National TV for 8 hours on Super Bowl Sunday. Speaking strictly personally, of course, the Puppy Bowl is harder to watch than soccer. Maybe even harder to watch than hockey. Maybe they could "glow" the fewmets so we can follow the action better. TurdTrax or something....
posted by umberto at 1:03 PM on January 7, 2009

owillis wrote "You do understand that poker... freaking poker... does better ratings on TV than hockey, right?"

Oh I know this. But that makes me more inclined to feel that the average person doesn't have the attention span to learn about things like line changes, penalty killing strategies, and the offside and icing rules. No, culturally when faced with a sport that has complicated* rules we'd rather just throw up our hands and go back to the spoonfed pap of crap like watching a group of people playing cards.

*Please note that by complicated I mean unfamiliar. Which is why baseball and football are popular, despite the amount of rules present in both games: More Americans are exposed to these sports while growing up, fewer are exposed to sports like hockey or cricket or soccer, so these games remain unfamiliar, foreign and off the radar despite their huge popularity in other countries. For the same reasons - familiarity - very few people outside of the US give two shits about American football.

I will state for the record though that I feel it is genetically impossible for one person to fully understand the rules for both cricket and baseball. Once you have learned one of them, your brain refuses to accept the other. I've tried, believe me...
posted by caution live frogs at 1:03 PM on January 7, 2009

The computer generated first-down line actually adds to the value of the broadcast by adding suspense. Fox's glowing puck detracted from the value of the broadcast because even though it was a technical marvel in terms of its complexity it looked stupid and cheap, and was unnecessary. Seriously, you can't see/follow the puck? Are Americans blind?
posted by The Card Cheat at 1:35 PM on January 7, 2009

It would never have occurred to me the camera itself was instrumented to make this possible.

SGI boxes for the actual compositing

Well, someone has to still buy computers made by SGI.
posted by Nelson at 1:59 PM on January 7, 2009

They weren't able to perfect it in time, but the holographic technology that CNN applied on Election Night is just as cool as 1st-and-Ten was a decade ago. They had to basically dumb it down (the technology) to get it to work well enough on air that night. Yes, yes, the result that night was rather underwhelming, but give it a year and you'll start seeing a lot more of it and then you will understand what they were really trying to pull off.
posted by intermod at 7:54 PM on January 7, 2009

Well, someone has to still buy computers made by SGI.

Coolest looking purple computers evah.
posted by rokusan at 10:04 PM on January 7, 2009