Somehow, this makes the Mushroom Kingdom even more trippy

May 31, 2011 2:52 PM

Depixelating Pixel Art: "Naïve upsampling of pixel art images leads to unsatisfactory results. Our algorithm extracts a smooth, resolution-independent vector representation from the image which is suitable for high-resolution display devices."

Supplemental material, with one-on-one and video comparisons.

posted by Diskeater (45 comments total)

I've seen this around for a few days now and my opinion can be summarized as:

1. This is really impressive
2. Good pixel art is still better
posted by 2bucksplus at 2:54 PM on May 31, 2011 [4 favorites]

So, how long till the de-voxel-ating Minecraft mod?
posted by Rhaomi at 2:55 PM on May 31, 2011 [2 favorites]

I've always wondered what Super Mario World would have looked like if it was a Newgrounds.com flash game from 2003.
posted by theodolite at 2:56 PM on May 31, 2011 [5 favorites]

I haven't messed with emulators in a while, but hopefully this will continue to get developed and adapted into the standard array of filters alongside 2xSAI and Eagle and whatnot. It could really breathe some new life into 8-/16-bit ga...HEY LET GO OF ME OW COME ON NOT THE FACE
posted by griphus at 3:03 PM on May 31, 2011 [4 favorites]

So, how long till the de-voxel-ating Minecraft mod?

not long
posted by Ad hominem at 3:04 PM on May 31, 2011 [2 favorites]

Yeah, I assume this can be hooked into a SNES emulator as some of those other algorithms were already running on ZSNES. That could make for some very interesting results.

I was really impressed by the Mario 2 pixel art here, because the pixel version is vastly superior to all of the attempts to depixelize it. That's pixel art at its absolute best.
posted by mek at 3:06 PM on May 31, 2011 [1 favorite]

I really rather like what it does to the Space Invader sprites.
posted by We had a deal, Kyle at 3:10 PM on May 31, 2011 [6 favorites]

I haven't messed with emulators in a while, but hopefully this will continue to get developed and adapted into the standard array of filters alongside 2xSAI and Eagle and whatnot.

That will require some significant optimizations. From the paper, the median calculation time was .62s per sprite. The worst case was 3.06s. And that on a modern 2.4GHz CPU. To render full screen at 30fps will require a speedup on the order of 100-fold. Perhaps it could be done by offloading it to a GPU and parallelizing it. The video examples were done by saving frames from an emulator and post-processing them.
posted by jedicus at 3:10 PM on May 31, 2011 [3 favorites]

Looking at their comparison page http://research.microsoft.com/en-us/um/people/kopf/pixelart/supplementary/multi_comparison.html

I must say I LOVE the results that the competing Vector Magic algorithm provides. It may not be a faithful representation, but I'd love to have those interpretive pieces hanging on my wall. I mean check out the dolphin and Yoshi.
posted by reformedjerk at 3:11 PM on May 31, 2011 [7 favorites]

It seems to work really well when the characters have black outlines; when they don't, like in the case of the Lost Vikings, the results aren't very good.
posted by Monday, stony Monday at 3:14 PM on May 31, 2011

The paper talks about that a little towards the end -- essentially their algorithm is looking for contrast between neighboring pixels, so it doesn't handle dithering between similar colors very well. (The example the paper uses is the marine from Doom.)
posted by We had a deal, Kyle at 3:17 PM on May 31, 2011

My initial visceral reaction to this was "Ugh! This is to classic 8-bit video games as colorization is to black and white movies!"

But it's pretty neat.
posted by usonian at 3:26 PM on May 31, 2011 [1 favorite]

Vector Magic's algorithm falls apart on small raster images, but it does so in terribly interesting ways. Love this new work on a technical level, though.
posted by anigbrowl at 3:41 PM on May 31, 2011

I must say I LOVE the results that the competing Vector Magic algorithm provides. It may not be a faithful representation, but I'd love to have those interpretive pieces hanging on my wall. I mean check out the dolphin and Yoshi.

Came here to say this, except that my favorites were Mario and Mario and Yoshi. And vector magic Bomber makes me very happy.
posted by zylocomotion at 3:49 PM on May 31, 2011 [2 favorites]

So they've essentially reverse-engineered livetrace? Good on 'em I guess.
posted by thsmchnekllsfascists at 3:56 PM on May 31, 2011

It didn't really consistently beat hq4x, IMO. There were some images that it did better with (that dolphin, boo) but did worse on lots too (it screws up the eyes in the 1up mushroom, and does a bad job with corners in general).

I don't really think it's possible to algorithmically get stuff "right" reliably - there just isn't enough visual information to tell if something's supposed to be curved or angled or rough or smooth, and you have to understand what it's trying to portray to make the call. I'd argue that's one reason why that low-res art can be so effective. There's room for you to fill in all those details with whatever it is your brain likes best.
posted by aubilenon at 3:59 PM on May 31, 2011

I came here to say how much I'd love to play an emulator with vector magic'ed sprites but it looks like I'm late to the party.
posted by aspo at 4:00 PM on May 31, 2011 [1 favorite]

Depixelation makes Mario sad.

_{Literally! Look at the Marios here!}
posted by ymgve at 4:01 PM on May 31, 2011 [1 favorite]

The Bomber with pink Powerpuff Girl Mary Janes tickles me to no end.
posted by darksasami at 4:03 PM on May 31, 2011

there just isn't enough visual information to tell if something's supposed to be curved or angled or rough or smooth, and you have to understand what it's trying to portray to make the call

It's not really that easy to do with ANYTHING, automatically or not. Low-res images are extremely difficult to trace by hand because there's no technically right answer. It's all an approximation of an image that somebody else made.
posted by thsmchnekllsfascists at 4:08 PM on May 31, 2011

Honestly, I like hq4x the best.
posted by clorox at 4:23 PM on May 31, 2011 [1 favorite]

This is cool from a look-what-someone-coded-at-SIGGRAPH perspective. Depending on what you're looking for, you might say that this algorithm blows away all of the other ones (though Vector Magic makes some pretty cool abstract art.) But where is this going to be applied? It seems like the authors were only interested in applying this to video game pixel art (that's pretty much the only place where it performs well, they say.) Can someone with more imagination than me suggest another application for this?

Otherwise, I start to lose my enthusiasm for the work, because it's pretty much just another scaling technique that makes pixel art look worse. No algorithm will ever "de-pixel" as well as the brain does. Because you're viewing pixel art with a head full of real world contextual knowledge, you often know when a 2x2 chunk of pixels is supposed to be a square or a circle. No reasonable amount of heuristics will make a smart algorithm that can interpret Yoshi's sprite better than a kid with a pencil. Computers just can't respect how cute his eyes are.
posted by MrFTBN at 4:24 PM on May 31, 2011

Computers just can't respect how cute his eyes are.

Famous last words.
posted by hellphish at 4:54 PM on May 31, 2011

I think these algorithms are great. I was a big fan of HQ4X and friends. A key difference here is the authors aren't worried with speed at all; part of the constraints of SuperEagle, HQ4X, etc is they can run at 30fps on 320x240 inputs. The paper doesn't have a lot of performance data but it's clearly a lot more work than the table lookups that are baked into game emulators now.

I wish they'd published some negative result images, the kinds of inputs it does poorly on. The Vector Magic examples they include on the supplemental page are interestingly awful.
posted by Nelson at 4:59 PM on May 31, 2011

Hah! Invader 1 > Vector Magic! > Star Trek Logo!
posted by SirOmega at 5:11 PM on May 31, 2011 [1 favorite]

I have a 16x image of the first Lost Viking now. Isn't this what internet is all about?
posted by ersatz at 5:21 PM on May 31, 2011

Oh, come on. They've been doing this on CSI for years.
posted by madcaptenor at 5:32 PM on May 31, 2011

But where is this going to be applied?

I think the point is not the application of this particular algorithm, but where it might lead development. The eventual goal, one presumes, is, as madcaptenor alludes, a CSI-style algorithm that will take low res digital images and "enhance."
posted by ocherdraco at 5:38 PM on May 31, 2011

I wish they'd published some negative result images, the kinds of inputs it does poorly on. The Vector Magic examples they include on the supplemental page are interestingly awful.

I dunno, a lot of the images on the supplementary page made it clear that this filter is not dealing with the pixel art as well as hq4x.

Also, although I've been a blasphemous user of filters in my emulator (and may continue to be one), when I came across this news earlier it prompted me to set my ZSNES settings to bilinear filter with 25% scanlines and that's it (along with slight aspect ratio distortion to make it a full-screen square in the middle of my widescreen monitor) for a nice TV-y effect that still caught some pixel sharpness. It felt right.
posted by Gnatcho at 5:51 PM on May 31, 2011

I think the point is not the application of this particular algorithm, but where it might lead development.

Well, yeah. That's pretty much what I meant.

...set my ZSNES settings to bilinear filter with 25% scanlines and that's it

Amen to that. Although you might want to give the newer ZSNES NTSC filter a try if you haven't already. It takes it a few steps further by attempting to simulate color bleeding, hue warping, and other nostalgia-inducing low res CRT artifacts.
posted by MrFTBN at 6:27 PM on May 31, 2011 [2 favorites]

That will require some significant optimizations. From the paper, the median calculation time was .62s per sprite. The worst case was 3.06s. And that on a modern 2.4GHz CPU. To render full screen at 30fps will require a speedup on the order of 100-fold. Perhaps it could be done by offloading it to a GPU and parallelizing it. The video examples were done by saving frames from an emulator and post-processing them.

Since they are sprites, all you need to do is pre-process the sprites at "boot" time. Sure, you might need another minute to boot your classic games, but it shouldn't impact actual runtime.
posted by b1tr0t at 6:30 PM on May 31, 2011

I wonder how well it works in a real setting. It's one thing to make things look good when the sprites are placed against a transparent or monochrome background. But ingame, you got a detailed background with no easy way to differentiate between character and background.

(You could of course do this by de-pixelizing sprites and background layers separately, but what are you going to do about characters consisting of multiple sprites? Or backgrounds with several layers?)
posted by ymgve at 7:06 PM on May 31, 2011

Since they are sprites, all you need to do is pre-process the sprites at "boot" time. Sure, you might need another minute to boot your classic games, but it shouldn't impact actual runtime.

Not everything in a 2D game is a sprite. And that would require some really low-level implementation in the emulator instead of being a comparatively simple output filter.
posted by jedicus at 7:13 PM on May 31, 2011

I wonder how well it works in a real setting.

The video was done by applying the algorithm to whole frames from an emulator. It seemed to work fine.
posted by jedicus at 7:14 PM on May 31, 2011

Yeah, I just discovered the video. Note, however, that Super Mario World has pretty "clean" graphics. They have also cropped out the score bar at the top, which might be one of the hardest parts to get right.

I wonder what the algorithm would do with more complex and detailed graphics.
posted by ymgve at 7:18 PM on May 31, 2011

I wonder whether this will make it into Windows before the last non-DPI-aware app falls out of use.
posted by flabdablet at 8:47 PM on May 31, 2011

I must say I LOVE the results that the competing Vector Magic algorithm provides. It may not be a faithful representation, but I'd love to have those interpretive pieces hanging on my wall.

Yeah, I was noticing Vector Magic and sometimes Live Trace produced the most interesting abstractions. I like them better than the rest, other than the original pixel art.
posted by krinklyfig at 8:50 PM on May 31, 2011

Although you might want to give the newer ZSNES NTSC filter a try if you haven't already. It takes it a few steps further by attempting to simulate color bleeding, hue warping, and other nostalgia-inducing low res CRT artifacts.

I hadn't seen that before, since I'm using the Linux SDL port and I figured out with some googling that I needed to turn OpenGL off for it to even be a visible choice. As for the filter itself, wow...I think my current settings help my nostalgic ideal a bit better than being confronted with an approximation closer to what it was really like :-). Also, it seems the option is unstable even with OpenGL turned off and causes lots of rendering errors (due to the netbook not being a powerhouse). But there are so many dials to play with in the NSTC settings!
posted by Gnatcho at 8:54 PM on May 31, 2011

Not everything in a 2D game is a sprite. And that would require some really low-level implementation in the emulator instead of being a comparatively simple output filter.

Sure, not everything 2D is a sprite, but everything in a NES game is a sprite.
posted by b1tr0t at 9:18 PM on May 31, 2011

You know, I actually *really really* like the "our version" of the Win3.1 mouse pointer.

A a naive observer, in general, it kinda looks like they used the old algorithm and tweaked edges, generically gave them more curve and less edge/point, tweaked/increased the gradient and colour-similarity threshold, and added "make round" rules for single pixels in colour isolation. I'm sure it's a lot more complicated than that.

Looks great on some things, doesn't really capture the original intent in others, ie., the broadsword (original).

Lots of "little stuff" but it works better than Hq4x mostly because it loses all of the "pixel looking" pixels.

Anyone ever used to design stuff in 8 bit? It took a lot more work/art to make a handful of pixels look like something than it does to be working in HD nowadays. I was lucky to work with 16 bits of colour over 8.

--

"Updating" old 8-bit games; extract all the art, run through this conversion. Hand-fix a few/as-much-as-is-necessary/practicable. Re-build game using the new art resources.
posted by porpoise at 9:24 PM on May 31, 2011

Can someone with more imagination than me suggest another application for this?

Software for vision-impaired people often magnifies the display, then has to smooth the results (Screen magnifier.) These approaches might be useful in smoothing the pixellated results better.
posted by alasdair at 12:18 AM on June 1, 2011

Zoom in. Enhance!
posted by bjrn at 3:25 AM on June 1, 2011

I love the plasticky feel of the "enhanced" Windows icons.
posted by brokkr at 7:02 AM on June 1, 2011

I love that Vector Magic turns "Help!" into something more closely approaching "Nerd!"

Also, count me among the folks who would love to have prints of some of those Vector Magic conversions on my wall.
posted by antifuse at 10:15 AM on June 1, 2011

Hah! Invader 1 > Vector Magic! > Star Trek Logo!

I noticed that series too... I think it would make a really good t-shirt.
posted by polymath at 11:22 PM on June 3, 2011