Arrival of a train
February 5, 2020 1:44 AM Subscribe
L'Arrivée d'un train en gare de La Ciotat (1896). L'Arrivée d'un train en gare de La Ciotat (2020). (via)
This is pretty incredible. I thought it was going to be a clip of a TGV just blowing by the platform, but that was a gag I remember from some French documentary made to celebrate the centenary of the Lumière brothers' invention. This upscaling is a super impressive feat that really makes the source material feel contemporary.
posted by St. Oops at 2:50 AM on February 5, 2020 [7 favorites]
posted by St. Oops at 2:50 AM on February 5, 2020 [7 favorites]
On first look of the 2020 I wondered if they were actors cast as lookalikes, on second I shuddered. I feel profoundly disturbed by these kids of projects, in the same way I disliked Peter Jackson’s movie colouring and narrating the First World War; it’s implying more information than is actually present in the film.
Neural nets and colouration and speculation are our best approximation of what was there but when we deal with the past we need to be aware of our loss too, that we’ll never know, not really. We need to be OK with that and be unprepared to take reconstruction as the past.
posted by Fiasco da Gama at 3:42 AM on February 5, 2020 [8 favorites]
Neural nets and colouration and speculation are our best approximation of what was there but when we deal with the past we need to be aware of our loss too, that we’ll never know, not really. We need to be OK with that and be unprepared to take reconstruction as the past.
posted by Fiasco da Gama at 3:42 AM on February 5, 2020 [8 favorites]
@Fiasco da Gama I think you're over-complicating it. There will never be more frames than the original 24 per second. No information is changing or being added, you can just see the information that was there more clearly.
posted by GallonOfAlan at 4:22 AM on February 5, 2020 [5 favorites]
posted by GallonOfAlan at 4:22 AM on February 5, 2020 [5 favorites]
these kids of projects
freudian slip?
I'm one of the class of people who see the 2020 as suffering from "soap opera effect" it doesn't look real to me because it looks too real. Like security camera footage of a bunch of actors in a costume drama. The same thing often happens to me if I watch a friends big TV, when I see a movie or commercial I feel like I can see the light stands just out of frame.
posted by Pembquist at 4:28 AM on February 5, 2020 [4 favorites]
freudian slip?
I'm one of the class of people who see the 2020 as suffering from "soap opera effect" it doesn't look real to me because it looks too real. Like security camera footage of a bunch of actors in a costume drama. The same thing often happens to me if I watch a friends big TV, when I see a movie or commercial I feel like I can see the light stands just out of frame.
posted by Pembquist at 4:28 AM on February 5, 2020 [4 favorites]
I am simultaneously fascinated by that and weirded out by it. It almost seems like it has fallen into the uncanny valley of too close to real without being real. I am not sure if that's because of the updating or if something contemporary filmed directly in 4k/60fps would feel the same way.
posted by jacquilynne at 4:40 AM on February 5, 2020 [1 favorite]
posted by jacquilynne at 4:40 AM on February 5, 2020 [1 favorite]
I don't disagree that colorization inserts a lot of assumptions into restored film, and I was not thrilled to hear Foley'd in sound effects in the 2020 version for the same reason. But I think it's perfectly fine to enhance and clarify the original footage as long as it is not imposing those additions. I thought this looked very good.
posted by briank at 4:43 AM on February 5, 2020 [3 favorites]
posted by briank at 4:43 AM on February 5, 2020 [3 favorites]
I love the fact that even though it's 1896 no one gives two shits about the camera except that one kid in the boater who takes a good hard look right at it.
posted by chavenet at 4:43 AM on February 5, 2020 [10 favorites]
posted by chavenet at 4:43 AM on February 5, 2020 [10 favorites]
About 10 years ago I chanced to be at someone's home while their rather expensive flat-panel entertainment system played one of the Pippi Longstocking films. This was in Germany so I think it was one of the 1960s releases.
I found myself mesmerised by it and wasn't sure why. It was only when I fiddled with the remote a bit that I realised some part of the DVD or TV system was inserting interstitial frames to smooth the motion out from the 24fps original to match the 50hz standard. Characters would turn their heads quickly and their hair would flip around in a way that seemed to me like live television rather than film.
Yes, there's a generational association between this framerate and either live broadcast or security camera footage. Historically those media had other quality problems, so we tended to prefer the soft effect of 24fps film to transport us into other worlds.
But there was also a generational bias toward black and white, once. Roger Ebert once wrote an essay defending B&W by saying that people dream in B&W so there must be something to it as a format. Once he published it, young people acted shocked that this was even a thing, while older people wondered how anyone could have gone their life without dreaming in B&W.
At some point we'll have a world where people whose media consumption wasn't all 4k 60fps are in the minority, and complaints about the unreality of these framerates will seem quaint.
But the stories of film school students who refuse to watch films in B&W still baffle and frustrate me!
posted by rum-soaked space hobo at 4:52 AM on February 5, 2020 [9 favorites]
I found myself mesmerised by it and wasn't sure why. It was only when I fiddled with the remote a bit that I realised some part of the DVD or TV system was inserting interstitial frames to smooth the motion out from the 24fps original to match the 50hz standard. Characters would turn their heads quickly and their hair would flip around in a way that seemed to me like live television rather than film.
Yes, there's a generational association between this framerate and either live broadcast or security camera footage. Historically those media had other quality problems, so we tended to prefer the soft effect of 24fps film to transport us into other worlds.
But there was also a generational bias toward black and white, once. Roger Ebert once wrote an essay defending B&W by saying that people dream in B&W so there must be something to it as a format. Once he published it, young people acted shocked that this was even a thing, while older people wondered how anyone could have gone their life without dreaming in B&W.
At some point we'll have a world where people whose media consumption wasn't all 4k 60fps are in the minority, and complaints about the unreality of these framerates will seem quaint.
But the stories of film school students who refuse to watch films in B&W still baffle and frustrate me!
posted by rum-soaked space hobo at 4:52 AM on February 5, 2020 [9 favorites]
GallonOfAlan: there are in fact more frames interpolated into the stream, using this tool.
posted by rum-soaked space hobo at 4:54 AM on February 5, 2020 [3 favorites]
posted by rum-soaked space hobo at 4:54 AM on February 5, 2020 [3 favorites]
Lest anyone think talk of colourisation is off-topic, they gave it a go with another tool. I'm not impressed with the results at all.
posted by rum-soaked space hobo at 4:57 AM on February 5, 2020 [3 favorites]
posted by rum-soaked space hobo at 4:57 AM on February 5, 2020 [3 favorites]
This looks like something for Erol Morris to comment on.
posted by From Bklyn at 4:57 AM on February 5, 2020 [2 favorites]
posted by From Bklyn at 4:57 AM on February 5, 2020 [2 favorites]
...part of the DVD or TV system was inserting interstitial frames to smooth the motion out from the 24fps original to match the 50hz standard...
This is most likely being done by the TV, it's colloquially called "soap opera effect" since soap operas are traditionally all 30fps videotape (vs the 24fps movie), although the technical term is 'motion smoothing'. You wouldn't think that 24 and 30 fps wouldn't be that different, but it's definitely noticeable, particularly since it's not 100% perfect to morph one frame into another to pull a brand new frame out of the two.
posted by AzraelBrown at 5:01 AM on February 5, 2020 [2 favorites]
This is most likely being done by the TV, it's colloquially called "soap opera effect" since soap operas are traditionally all 30fps videotape (vs the 24fps movie), although the technical term is 'motion smoothing'. You wouldn't think that 24 and 30 fps wouldn't be that different, but it's definitely noticeable, particularly since it's not 100% perfect to morph one frame into another to pull a brand new frame out of the two.
posted by AzraelBrown at 5:01 AM on February 5, 2020 [2 favorites]
I think the faces look too smooth in the cleaned-up version-- almost mask-like.
posted by Nancy Lebovitz at 5:10 AM on February 5, 2020
posted by Nancy Lebovitz at 5:10 AM on February 5, 2020
The original wasn't necessarily shot at 24 fps or any consistent speed. Silent cameras were usually hand-cranked and speeds were generally around 16 fps but varied all over the place.
posted by octothorpe at 5:21 AM on February 5, 2020 [9 favorites]
posted by octothorpe at 5:21 AM on February 5, 2020 [9 favorites]
I was astounded by the amount detail that seemed to magically appear in the 2020 version but when they show a comparison between the print they stated with and the upscaled version, you can see their source is a WAY better print than the first link.
posted by bonobothegreat at 5:26 AM on February 5, 2020 [11 favorites]
posted by bonobothegreat at 5:26 AM on February 5, 2020 [11 favorites]
A good modern 35mm negative can have very close to 4k resolution but I don't know what shape the existing copies of the Lumière film are in. In any case, this new version didn't start from a new scan but an old lower-resolution scan.
posted by octothorpe at 5:43 AM on February 5, 2020 [1 favorite]
posted by octothorpe at 5:43 AM on February 5, 2020 [1 favorite]
Some of the film is so crystal-clear it's spooky, but then there are odd shifts where, I'm guessing, there aren't frames capturing the motion in the original, so everybody kind of leaps into their next position, which makes it....even more spooky. Still, seeing everybody in this famous clip very clearly, even if much of it is interpolation, is pretty cool.
posted by xingcat at 5:53 AM on February 5, 2020 [2 favorites]
posted by xingcat at 5:53 AM on February 5, 2020 [2 favorites]
I took a freeze frame of the HD original (not the vimeo link) and compared it with the 2020 version at the 30 second mark.
You can see in this that the digital sharpening has done a great job with all the straight edges, the train doors, the hats, the patterned blouse that the woman is wearing, all look much clearer and with better contrast.
But now look at the face of the child, something which doesnt have any straight edges for the algorithm to grab onto and what you see is a horrorshow!
This is why I still shoot film.
posted by Lanark at 5:59 AM on February 5, 2020 [13 favorites]
You can see in this that the digital sharpening has done a great job with all the straight edges, the train doors, the hats, the patterned blouse that the woman is wearing, all look much clearer and with better contrast.
But now look at the face of the child, something which doesnt have any straight edges for the algorithm to grab onto and what you see is a horrorshow!
This is why I still shoot film.
posted by Lanark at 5:59 AM on February 5, 2020 [13 favorites]
It looked to me like it was interpreting women with veils as faceless beings with white chinstrap beards.
posted by moonmilk at 6:01 AM on February 5, 2020 [3 favorites]
posted by moonmilk at 6:01 AM on February 5, 2020 [3 favorites]
Putting framerate issues aside, I'm mostly just annoyed that they cropped the original to make it fit a 16:9 aspect ratio. They're literally ruining the composition and cropping out content.
posted by schmod at 6:38 AM on February 5, 2020 [2 favorites]
posted by schmod at 6:38 AM on February 5, 2020 [2 favorites]
Every episode of CSI:whatever just took a big step toward non-fiction.
"Zoom in. Enhance."
posted by Frayed Knot at 6:43 AM on February 5, 2020
"Zoom in. Enhance."
posted by Frayed Knot at 6:43 AM on February 5, 2020
*dives out of the way*
posted by scratch at 6:52 AM on February 5, 2020 [1 favorite]
posted by scratch at 6:52 AM on February 5, 2020 [1 favorite]
Every episode of CSI:whatever just took a big step toward non-fiction....but in a completely terrifying way. The neural network isn't filling in details by reconstructing the "real image" – it's filling them in by substituting similar-looking features from its training set.
Just hope that, if you're accused of a crime, your face isn't part of the police department's training set.
posted by schmod at 6:52 AM on February 5, 2020 [3 favorites]
Yeah, the presentation of this post is a little misleading. The initial Vimeo link is much lower quality than the actual source used for the improved version.
posted by Reverend John at 6:55 AM on February 5, 2020 [1 favorite]
posted by Reverend John at 6:55 AM on February 5, 2020 [1 favorite]
Lanark: Or, it could be that that couple has adopted a chimpanzee. We have no way of knowing for sure.
posted by SansPoint at 7:03 AM on February 5, 2020 [3 favorites]
posted by SansPoint at 7:03 AM on February 5, 2020 [3 favorites]
(Hmmm, though possibly that "actual source" was itself a cleanup job of the video from the first link of the post ... so I might be mistaken in calling the presentation misleading).
posted by Reverend John at 7:08 AM on February 5, 2020
posted by Reverend John at 7:08 AM on February 5, 2020
Here's a better look at the details of what was done, from Ars Technica. There's a fair amount in the comments which is new to me, especially this one which covers why frame rates make such a difference.
posted by ambrosen at 7:08 AM on February 5, 2020 [2 favorites]
posted by ambrosen at 7:08 AM on February 5, 2020 [2 favorites]
Yes, there's a generational association between this framerate and either live broadcast or security camera footage. Historically those media had other quality problems, so we tended to prefer the soft effect of 24fps film to transport us into other worlds.
This weekend I had to watch the superbowl on the fox app on my roku - so the streamed cable feed - after years of watching the digital broadcast version (The FCC mandated broadcast repack resulted in Fox32 in Chicago "temporarily" switching to their auxiliary which is on top the John Hancock building rather than the Willis Tower - resulting in many many lakefront condos blocking my broadcast signal despite being 16 floors up myself). It was awful. I thought my eyes were failing. I kept blinking to see if I could bring it into focus. It felt so incredibly wrong. It is very weird how we are trained to 'see'.
posted by srboisvert at 7:12 AM on February 5, 2020 [1 favorite]
This weekend I had to watch the superbowl on the fox app on my roku - so the streamed cable feed - after years of watching the digital broadcast version (The FCC mandated broadcast repack resulted in Fox32 in Chicago "temporarily" switching to their auxiliary which is on top the John Hancock building rather than the Willis Tower - resulting in many many lakefront condos blocking my broadcast signal despite being 16 floors up myself). It was awful. I thought my eyes were failing. I kept blinking to see if I could bring it into focus. It felt so incredibly wrong. It is very weird how we are trained to 'see'.
posted by srboisvert at 7:12 AM on February 5, 2020 [1 favorite]
I would like to see this train arrive through Deepdream: fish-dog heads and eyes sprouting everywhere.
posted by scruss at 7:15 AM on February 5, 2020 [5 favorites]
posted by scruss at 7:15 AM on February 5, 2020 [5 favorites]
To my eye this seems to be operating on the edge conditions between forms and along shadow edges within the image, basically losing a lot of "blur" information ini favor of a kind of binary "1" or "0" over any gradient. This might explain my feeling of seeing something more uncanny than real. The sharpened edges of the train doors and in the shadows on peoples faces not resulting in "realism" but instead a kind of angular dreaminess within an infinite depth of field. It's not so much more "naturalistic" than the source as it shifted into a kind of different perceptual space that has the trappings of realism but without something essential to creating realistic effect. There's a lot to think about here...
posted by hilberseimer at 7:16 AM on February 5, 2020 [1 favorite]
posted by hilberseimer at 7:16 AM on February 5, 2020 [1 favorite]
The neural network isn't filling in details by reconstructing the "real image" – it's filling them in by substituting similar-looking features from its training set.
Yes, exactly. Pixels are information, and you can't get information from nothing. Machine learning (e.g. via neural network) basically smuggles in the "missing" information by borrowing some from the training set. That's it, there's no magic.
Now, the technology is both cool and useful, but it's extremely important -- like, the future of human civilization depends on it -- that people understand that the information added via this kind of processing absolutely is not present in the source material. The end result can never be anything more than a really good guess at what the camera did not record.
We don't even need to imagine the possible negative consequences of failing to understand the distinction, because we already have these godforsaken ML algorithms ruining the actual lives of actual people. (Previously & previously.)
I love looking at the results of stuff like this, but the way it's often presented to the public is dangerously misleading.
posted by tobascodagama at 8:03 AM on February 5, 2020 [12 favorites]
Yes, exactly. Pixels are information, and you can't get information from nothing. Machine learning (e.g. via neural network) basically smuggles in the "missing" information by borrowing some from the training set. That's it, there's no magic.
Now, the technology is both cool and useful, but it's extremely important -- like, the future of human civilization depends on it -- that people understand that the information added via this kind of processing absolutely is not present in the source material. The end result can never be anything more than a really good guess at what the camera did not record.
We don't even need to imagine the possible negative consequences of failing to understand the distinction, because we already have these godforsaken ML algorithms ruining the actual lives of actual people. (Previously & previously.)
I love looking at the results of stuff like this, but the way it's often presented to the public is dangerously misleading.
posted by tobascodagama at 8:03 AM on February 5, 2020 [12 favorites]
> schmod:
> The neural network isn't filling in details by reconstructing the "real image" – it's
> filling them in by substituting similar-looking features from its training set."
The second part is true only in a *very* general sense. And in that case, our brains do the same thing when we watch the low-res version. Which, yeah, doesn't exactly make this (or people) any less scary as a possible forensic tool btw...
posted by kleinsteradikaleminderheit at 8:12 AM on February 5, 2020 [1 favorite]
> The neural network isn't filling in details by reconstructing the "real image" – it's
> filling them in by substituting similar-looking features from its training set."
The second part is true only in a *very* general sense. And in that case, our brains do the same thing when we watch the low-res version. Which, yeah, doesn't exactly make this (or people) any less scary as a possible forensic tool btw...
posted by kleinsteradikaleminderheit at 8:12 AM on February 5, 2020 [1 favorite]
I'd like to see this done with some old baseball footage. You know, the important stuff.
posted by zzazazz at 8:24 AM on February 5, 2020 [1 favorite]
posted by zzazazz at 8:24 AM on February 5, 2020 [1 favorite]
I've been playing with rendering weather satellite imagery as video, and I have yet to come up with a better method for interpolating between source frames than just doing a straight-up blend between the real frames either side of the interpolations. I get 60fps results that do look enough less distractingly juddery than the 12fps inputs I'm starting with to make the exercise worthwhile, but there are still a lot of noticeable 12fps artifacts just because every fifth frame is not a blend.
All the "smart" motion interpolation tools I've played with so far have yielded much much worse results. Clouds often don't have well-defined edges and often do have variable transparency, and these two facts between them seem to play merry hell with the assumptions made by motion interpolators about what and where the moving objects are. At best the results have been no better than my simple-minded blending interpolations; at worst I get little bits of the underlying landscape waving about in the wind in a highly disconcerting way.
Once the depth-aware stuff makes it into mainline ffmpeg I'll have a go with that, but I'm not expecting great results.
posted by flabdablet at 8:33 AM on February 5, 2020 [3 favorites]
All the "smart" motion interpolation tools I've played with so far have yielded much much worse results. Clouds often don't have well-defined edges and often do have variable transparency, and these two facts between them seem to play merry hell with the assumptions made by motion interpolators about what and where the moving objects are. At best the results have been no better than my simple-minded blending interpolations; at worst I get little bits of the underlying landscape waving about in the wind in a highly disconcerting way.
Once the depth-aware stuff makes it into mainline ffmpeg I'll have a go with that, but I'm not expecting great results.
posted by flabdablet at 8:33 AM on February 5, 2020 [3 favorites]
I love how the 2020 version starts with flickering film and scratches, a truly cheesy digital effect which everybody uses to denote “it’s old.” Not impressed.
posted by njohnson23 at 9:04 AM on February 5, 2020 [2 favorites]
posted by njohnson23 at 9:04 AM on February 5, 2020 [2 favorites]
I’m interested in conditions where these cliches aren’t cheesy, just because they always seem to be. Music production can get away with it more I think, but that might be wrong.
posted by hilberseimer at 9:07 AM on February 5, 2020
posted by hilberseimer at 9:07 AM on February 5, 2020
Music production getting away with it
posted by flabdablet at 9:33 AM on February 5, 2020 [1 favorite]
posted by flabdablet at 9:33 AM on February 5, 2020 [1 favorite]
Weather interpolation:
You might need to train something custom for your domain. "The map should stay still while the clouds move" is an important domain detail that a general purpose video interpolation model won't have much respect for.
posted by kaibutsu at 9:41 AM on February 5, 2020 [1 favorite]
You might need to train something custom for your domain. "The map should stay still while the clouds move" is an important domain detail that a general purpose video interpolation model won't have much respect for.
posted by kaibutsu at 9:41 AM on February 5, 2020 [1 favorite]
I kind of expect that given the variable lighting that clouds and the day/night cycle impose on the underlying landscape, any interpolator is always going to have a hard time figuring out how much of any given pixel to assign to the moving thing and how much to the static thing.
My current line of experimentation involves blending more than two input frames to try to damp down the 12fps artifacts. This might end up generating 6fps artifacts instead or as well, and it will certainly cost detail on fast-moving cloud, but if I want to look at detail that's what the original imagery is for; I don't mind a bit of motion blur if it makes the motion look plausibly and non-distractingly fluid.
posted by flabdablet at 10:36 AM on February 5, 2020
My current line of experimentation involves blending more than two input frames to try to damp down the 12fps artifacts. This might end up generating 6fps artifacts instead or as well, and it will certainly cost detail on fast-moving cloud, but if I want to look at detail that's what the original imagery is for; I don't mind a bit of motion blur if it makes the motion look plausibly and non-distractingly fluid.
posted by flabdablet at 10:36 AM on February 5, 2020
For the 2020 version, I was expecting a video of a modern train arriving at the same station, shot from the same angle. For some reason, nobody seems to have done that kind of juxtaposition.
posted by beagle at 12:41 PM on February 5, 2020 [2 favorites]
posted by beagle at 12:41 PM on February 5, 2020 [2 favorites]
That is also what I expected when I first clicked - the TGV rocketing in or through.
posted by jacquilynne at 2:53 PM on February 5, 2020
posted by jacquilynne at 2:53 PM on February 5, 2020
There’s a 2016 version with a modern train here. Oddly on my devices it plays portrait and reversed. I’m not sure if this is biting commentary or just a tech glitch.
posted by grahamparks at 4:21 PM on February 5, 2020 [1 favorite]
posted by grahamparks at 4:21 PM on February 5, 2020 [1 favorite]
No French train arrival is complete without this sound. In 1896 it would have been played live by a small chamber orchestra. (As created by these guys)
posted by rongorongo at 1:53 AM on February 6, 2020
posted by rongorongo at 1:53 AM on February 6, 2020
No information is changing or being added, you can just see the information that was there more clearly.
Sorry, this is not correct. Many pixels and frames are being added, which constitute information which not present in the original - it's "guessed" or interpolated. That's what "upscaling" is. The original footage was not 60FPS but closer to 15, so 3/4 of the frames had to be synthesized to create this. If you download the files, you will see that the original movie file is 6.1MB while the reconstruction is 206MB - so that's about 200 megabytes of added information. Blurring/lack of image quality in the original means that information content is low (not captured due to technical limitations of the equipment at the time, and further information loss due to degradation over time) - this lost information is not somehow "hidden", the upscaling process simply makes it up (using maximum-likelihood estimates and other statistical techniques).
posted by crazy_yeti at 8:19 AM on February 6, 2020 [3 favorites]
Sorry, this is not correct. Many pixels and frames are being added, which constitute information which not present in the original - it's "guessed" or interpolated. That's what "upscaling" is. The original footage was not 60FPS but closer to 15, so 3/4 of the frames had to be synthesized to create this. If you download the files, you will see that the original movie file is 6.1MB while the reconstruction is 206MB - so that's about 200 megabytes of added information. Blurring/lack of image quality in the original means that information content is low (not captured due to technical limitations of the equipment at the time, and further information loss due to degradation over time) - this lost information is not somehow "hidden", the upscaling process simply makes it up (using maximum-likelihood estimates and other statistical techniques).
posted by crazy_yeti at 8:19 AM on February 6, 2020 [3 favorites]
This is a completely unreasonable pie-in-the-sky thing, but I do wish that information theory were more widely taught. It's only going to get more important the products of machine learning become more intertwined in people's lives.
posted by tobascodagama at 9:08 AM on February 6, 2020 [2 favorites]
posted by tobascodagama at 9:08 AM on February 6, 2020 [2 favorites]
I think I like the already improved source video better than the enhanced one. The Gigapixel AI program that Ars Technica said was used is good though. I use it regularly at work and if the original is not too bad it usually ends up looking much better at double size or more than the original.
posted by blue shadows at 11:33 AM on February 6, 2020
posted by blue shadows at 11:33 AM on February 6, 2020
Denis Shiryaev has posted another great NN enhanced video: footage of New York Ciity shot in 1911. On the one hand this is the distant past - and on the other the place was full of electric trains and electric cars.
posted by rongorongo at 4:36 AM on February 27, 2020 [2 favorites]
posted by rongorongo at 4:36 AM on February 27, 2020 [2 favorites]
« Older Betty White Gets The Giggles | In Praise of (the Fashion of) Tintin Newer »
This thread has been archived and is closed to new comments
posted by fairmettle at 2:42 AM on February 5, 2020 [6 favorites]