It's like Greasemonkey, but for your GUI
April 2, 2010 7:45 AM Subscribe
Have you ever wanted to change the functionality of the GUI of a program that you didn't have the source code for? Prefab is a tool that was made to allow you to do exactly that.
In the creators' own words (pdf):
In the creators' own words (pdf):
Current methods for implementing graphical user interfaces create fundamental challenges for HCI research and practice. Researchers are often unable to demonstrate or evaluate new techniques beyond small toy applications, and practitioners are often unable to adopt methods from the literature in new and existing applications. This position statement examines a vision in which anybody can modify any GUI of any application, similar to a scenario where every GUI of every application is open-source. We are currently working to enable this vision through our development of Prefab, using pixel-based interpretation of GUIs to enable modification of those GUIs without any cooperation from the underlying application. We see participation in the FLOSS HCI workshop as valuable in at least two regards. First, fully realizing this vision will likely require a community-based approach, so we are interested in Prefab as a platform for collaboration between HCI researchers and the FLOSS community. Second, enabling arbitrary modification of any GUI would seem to blur many current distinctions between open and closed applications, introducing new research questions and further magnifying the importance of the workshop’s focus.
I know this makes me a bad person, but I started giggling about 5 seconds after the video started. Super cereal, guys!
posted by hincandenza at 7:54 AM on April 2, 2010
posted by hincandenza at 7:54 AM on April 2, 2010
lots of predictable stupidity in the comments on this P-I article, including CLI-guy, MS-Lover, Open Source dude, and HCI-hater. It's like troll bingo!
posted by mwhybark at 7:58 AM on April 2, 2010 [3 favorites]
posted by mwhybark at 7:58 AM on April 2, 2010 [3 favorites]
But that was very cool- and clever, to make it something so last-second superficial that it's inherently cross-platform and extendable, as ArgentCorvid mentioned. I'm kind of curious if an entirely new interface could be crafted; it certainly seems that way.
posted by hincandenza at 8:00 AM on April 2, 2010
posted by hincandenza at 8:00 AM on April 2, 2010
I would be astonished if this has any practical end-user application, but I don't think that's what it's for.
It allows UI researchers to take a program like Media Player or Internet Explorer, change how the UI works, and then test the results on end users.
Then the end users are using a program they know, and you can remove a significant source of uncertainty from your UI research - are you testing the new UI features, such as those demonstrated in the video, or are you testing how much users can handle a new program per se? And for that it'll be brilliant.
Otherwise all you're testing is how much your test program is like the program the user is used to.
posted by alasdair at 8:24 AM on April 2, 2010 [2 favorites]
It allows UI researchers to take a program like Media Player or Internet Explorer, change how the UI works, and then test the results on end users.
Then the end users are using a program they know, and you can remove a significant source of uncertainty from your UI research - are you testing the new UI features, such as those demonstrated in the video, or are you testing how much users can handle a new program per se? And for that it'll be brilliant.
Otherwise all you're testing is how much your test program is like the program the user is used to.
posted by alasdair at 8:24 AM on April 2, 2010 [2 favorites]
I seriously want this, it looks incredibly useful. I don't know if it's because I've recently got a new mouse, but I think automatically selecting the nearest control would make using a mouse a hell of a lot more convenient. I think it would really speed up using a desktop not to have to be so precise with your hand movements, I really think the level of precision needed currently is a pain. So hopefully this will be downloadable soon, or someone will hack the real thing into a Firefox extension or something. (I'm talking about the first few, mouse-based examples in the video, not the preview-spectrum thing).
posted by rubber duck at 8:27 AM on April 2, 2010 [1 favorite]
posted by rubber duck at 8:27 AM on April 2, 2010 [1 favorite]
To merge threads, so I can make a UI for dwarf fortress?
posted by a robot made out of meat at 8:28 AM on April 2, 2010 [6 favorites]
posted by a robot made out of meat at 8:28 AM on April 2, 2010 [6 favorites]
I really appreciated the comment by "david z" on one of the blogs linked from the project page. I keep encountering an attitude in the business world that I don't understand, a profound mistrust of all things open-source. What's not to trust? Everything OSS does is transparent. It's the black box of closed-source software we should be wary of.
At any rate, suggesting that Prefab is able to turn closed-source software into open-source is not helpful. It only confuses the issue. Better to describe it as an ad hoc, aftermarket API, or context-aware screen-scraper or something. That said, it could be some useful technology. It wasn't clear to me how it runs. If it's a background process that can identify windows when they appear onscreen and track their movement if you drag them, I can see massive value in something like that. From what the video presented it didn't seem quite that automatic. I haven't read the PDF yet.
posted by The Winsome Parker Lewis at 8:33 AM on April 2, 2010
At any rate, suggesting that Prefab is able to turn closed-source software into open-source is not helpful. It only confuses the issue. Better to describe it as an ad hoc, aftermarket API, or context-aware screen-scraper or something. That said, it could be some useful technology. It wasn't clear to me how it runs. If it's a background process that can identify windows when they appear onscreen and track their movement if you drag them, I can see massive value in something like that. From what the video presented it didn't seem quite that automatic. I haven't read the PDF yet.
posted by The Winsome Parker Lewis at 8:33 AM on April 2, 2010
This would be great for turning a conventional app into a touch-screen app.
posted by bhnyc at 8:38 AM on April 2, 2010 [3 favorites]
posted by bhnyc at 8:38 AM on April 2, 2010 [3 favorites]
I would be astonished if this has any practical end-user application, but I don't think that's what it's for.
Just the part where it can automate UI interaction would be a big plus for me personally. I have a media theater PC that I use a remote to control, but that only really works well for applications that have keyboard shortcuts for everything I want to do. It would be neat to be able to, say, control YouTube videos with the mouse or even re-organize the whole OS around a UI that makes more sense for using a remote.
posted by burnmp3s at 8:39 AM on April 2, 2010
Just the part where it can automate UI interaction would be a big plus for me personally. I have a media theater PC that I use a remote to control, but that only really works well for applications that have keyboard shortcuts for everything I want to do. It would be neat to be able to, say, control YouTube videos with the mouse or even re-organize the whole OS around a UI that makes more sense for using a remote.
posted by burnmp3s at 8:39 AM on April 2, 2010
I would be astonished if this has any practical end-user application, but I don't think that's what it's for.
I would be astonished if it doesn't. I currently have the vimperator plugin installed, which completely redefines the interface to firefox (makes it entirely controllable by keyboard). It does so entirely through javascript. This kind of sucks. It's nigh impossible to interact with a page while it's loading because firefox just doesn't care to process your keyboard input at that time, and vimperator becomes frozen. Meanwhile, clicking the x will kill that page just fine. Flash based websites are impossible to interact with without resorting to the mouse. Implementing some of the features of vimperator according to this design would be an awesome boon to me.
posted by Bobicus at 8:41 AM on April 2, 2010 [2 favorites]
I would be astonished if it doesn't. I currently have the vimperator plugin installed, which completely redefines the interface to firefox (makes it entirely controllable by keyboard). It does so entirely through javascript. This kind of sucks. It's nigh impossible to interact with a page while it's loading because firefox just doesn't care to process your keyboard input at that time, and vimperator becomes frozen. Meanwhile, clicking the x will kill that page just fine. Flash based websites are impossible to interact with without resorting to the mouse. Implementing some of the features of vimperator according to this design would be an awesome boon to me.
posted by Bobicus at 8:41 AM on April 2, 2010 [2 favorites]
We are currently working to enable this vision through our development of Prefab, using pixel-based interpretation of GUIs to enable modification of those GUIs without any cooperation from the underlying application.
Wow, that's pretty impressive.
posted by delmoi at 8:53 AM on April 2, 2010
Wow, that's pretty impressive.
posted by delmoi at 8:53 AM on April 2, 2010
rubber duck: maybe you need to get a trackball, instead. My computer life was infinitely improved when I switched to a thumb-operated trackball. Precision became a non-issue, and the savings in desk-space (using a device which itself does not move) is wonderful.
/derail
posted by hippybear at 8:56 AM on April 2, 2010 [1 favorite]
/derail
posted by hippybear at 8:56 AM on April 2, 2010 [1 favorite]
rubber duck: but I think automatically selecting the nearest control would make using a mouse a hell of a lot more convenient.
You should try an OS that allows focus-follows-mouse (usually Linux; Ubuntu does, Mac OS appears not to). I tend to have it off most of the time, but on some projects I do enable it and it is very convenient.
The other alternative is software that actually jumps the mouse cursor to the nearest control instead of letting you have pixel-level control over where you move the mouse. I tried software that did this and I have to say I hated it. It was fine for relatively closely-grouped control sets, but if you have to move your mouse cursor more than a few inches, it feels like you've got a broken mouse -- the cursor jumps around and resists mouse movement.
The closest thing I can think to approximate it would be to turn on all the automatic snapping features in a drawing package and work that way for a while. Like "smart guides" in Adobe software -- it's nice when you need it, but man does it get annoying sometimes.
posted by circular at 9:10 AM on April 2, 2010
You should try an OS that allows focus-follows-mouse (usually Linux; Ubuntu does, Mac OS appears not to). I tend to have it off most of the time, but on some projects I do enable it and it is very convenient.
The other alternative is software that actually jumps the mouse cursor to the nearest control instead of letting you have pixel-level control over where you move the mouse. I tried software that did this and I have to say I hated it. It was fine for relatively closely-grouped control sets, but if you have to move your mouse cursor more than a few inches, it feels like you've got a broken mouse -- the cursor jumps around and resists mouse movement.
The closest thing I can think to approximate it would be to turn on all the automatic snapping features in a drawing package and work that way for a while. Like "smart guides" in Adobe software -- it's nice when you need it, but man does it get annoying sometimes.
posted by circular at 9:10 AM on April 2, 2010
This makes sense for doing HCI research on real applications, rather that toys, but the latency that it adds is going to make a tough sell to normal users. An important thing to realize is that the modification of the interface is going to take a substantial amount of code itself, although some templates could be made for commonly used applications.
And what's with the "Bubble Cursor"? It's not their invention, but it looks really annoying to me.
posted by demiurge at 9:13 AM on April 2, 2010
And what's with the "Bubble Cursor"? It's not their invention, but it looks really annoying to me.
posted by demiurge at 9:13 AM on April 2, 2010
They are hitting the wrong approach.
posted by adipocere at 9:15 AM on April 2, 2010 [2 favorites]
command line:Expect :: gui:SIKULIposted by adipocere at 9:15 AM on April 2, 2010 [2 favorites]
I was also going to mention sikuli for automation, but the autorecognition of widgets in this project is very nice.
posted by Brent Parker at 9:27 AM on April 2, 2010
posted by Brent Parker at 9:27 AM on April 2, 2010
@1:18 did he just say "pacifically" rather than "specifically"?
posted by milnak at 10:05 AM on April 2, 2010
posted by milnak at 10:05 AM on April 2, 2010
You should try an OS that allows focus-follows-mouse
No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
posted by GuyZero at 10:15 AM on April 2, 2010 [2 favorites]
No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
posted by GuyZero at 10:15 AM on April 2, 2010 [2 favorites]
No one's used it since mid-90's X window managers for a reason.
...except for my CS professor partner who rants about any OS which doesn't have that feature.
posted by hippybear at 10:25 AM on April 2, 2010
...except for my CS professor partner who rants about any OS which doesn't have that feature.
posted by hippybear at 10:25 AM on April 2, 2010
If you have a 40-pound mouse it's pretty usable I guess.
posted by GuyZero at 10:36 AM on April 2, 2010
posted by GuyZero at 10:36 AM on April 2, 2010
> No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
It was indeed there in the 90s, but it's still right there in the control panel on my desktop machine. Did you use it with raise-windows enabled? Cuz I'm not a big fan of that.
posted by circular at 10:48 AM on April 2, 2010
It was indeed there in the 90s, but it's still right there in the control panel on my desktop machine. Did you use it with raise-windows enabled? Cuz I'm not a big fan of that.
posted by circular at 10:48 AM on April 2, 2010
I use focus follows mouse all the time. Love it. It's like the first thing I do when I configure a new window manager. I am not a CS person either; I'm a librarian with an English degree.
posted by the dief at 10:52 AM on April 2, 2010 [1 favorite]
posted by the dief at 10:52 AM on April 2, 2010 [1 favorite]
Does it work in Dwarf Fortress?
posted by infinitywaltz at 11:19 AM on April 2, 2010 [1 favorite]
posted by infinitywaltz at 11:19 AM on April 2, 2010 [1 favorite]
Not to contribute to the derail too much but Focus Follows Mouse is available in Windows (at least XP) through Microsoft's TweakUI tool.
posted by ChrisHartley at 11:20 AM on April 2, 2010
posted by ChrisHartley at 11:20 AM on April 2, 2010
To continue the focus-follows-mouse derail (the one true way!), you can enable it in OS X's Terminal.app by running (in a terminal window, of course):
posted by autopilot at 11:43 AM on April 2, 2010 [1 favorite]
defaults write com.apple.Terminal FocusFollowsMouse -string YESSince I only run two applications regularly (Firefox and Terminal) this solves 50% of my GUI problems on my Macbook.
posted by autopilot at 11:43 AM on April 2, 2010 [1 favorite]
The only way this thread could be more wrong is people showed up and started deriding paper in favour of clay tablets. Do you people have crippled index fingers or something?
posted by GuyZero at 11:46 AM on April 2, 2010
posted by GuyZero at 11:46 AM on April 2, 2010
The mouse gain modification seems like it would be incredibly annoying in practical use. I would hate to have my mouse randomly slow down when sweeping across a window with lots of widgets.
posted by archagon at 12:02 PM on April 2, 2010
posted by archagon at 12:02 PM on April 2, 2010
The two ideas why said it could work on at first, that bubble curser thing and the one that slowed down over buttons, looked kind of like they would really be of no use to me. The last one, where they showed how you moved sliders and tabs over time, looks really really cool. WANT.
posted by The Devil Tesla at 12:28 PM on April 2, 2010
posted by The Devil Tesla at 12:28 PM on April 2, 2010
See, the thing is, I work in assistive technology: providing alternative mechanisms to the standard keyboard/mouse interface for blind people, Braille users, people who have no arms, people who can't stop their hands shaking - that sort of thing. This requires mechanisms that, yes, identify control structures and permit different controls and approaches.
And it's a pain. Different interfaces (.Net, Windows API, GTK, iTunes until recently, Java) different visual styles (XP style, Classic style, Aero on or off, High Visibility) different versions (Skype version 1 versus Skype version 2 - like, a completely different UI on a complete different program) different contexts and re-uses of existing components (hypertext links suddenly becoming control buttons in client applications, plain text areas getting onclick activity, mouseovers on things that "shouldn't" have mouseovers). It's a really really heterogeneous world out there, and it's so very hard to make something like their special mouse operations work consistently over time and over different applications.
That's why I think this'll be great for testing UI theories, and not so useful for actual day-to-day use. But if I'm wrong, hey, there's another approach that'll work well for assistive technology too, so this is win-win for me!
An aside: the control size for a checkbox is actually [the checkbox itself] + [the size of the text of the checkbox] - this makes it far easier to hit. You can see this in any Windows app (I'm going to assume other OSs are the same) and in webpages using the LABEL attribute correctly. So they need to update their code a bit there. Unless it's deliberate, of course, I suddenly think - might be clearer to JUST point to the checkbox, and the user can see the text. Ah, that makes sense.
posted by alasdair at 3:07 PM on April 2, 2010 [2 favorites]
And it's a pain. Different interfaces (.Net, Windows API, GTK, iTunes until recently, Java) different visual styles (XP style, Classic style, Aero on or off, High Visibility) different versions (Skype version 1 versus Skype version 2 - like, a completely different UI on a complete different program) different contexts and re-uses of existing components (hypertext links suddenly becoming control buttons in client applications, plain text areas getting onclick activity, mouseovers on things that "shouldn't" have mouseovers). It's a really really heterogeneous world out there, and it's so very hard to make something like their special mouse operations work consistently over time and over different applications.
That's why I think this'll be great for testing UI theories, and not so useful for actual day-to-day use. But if I'm wrong, hey, there's another approach that'll work well for assistive technology too, so this is win-win for me!
An aside: the control size for a checkbox is actually [the checkbox itself] + [the size of the text of the checkbox] - this makes it far easier to hit. You can see this in any Windows app (I'm going to assume other OSs are the same) and in webpages using the LABEL attribute correctly. So they need to update their code a bit there. Unless it's deliberate, of course, I suddenly think - might be clearer to JUST point to the checkbox, and the user can see the text. Ah, that makes sense.
posted by alasdair at 3:07 PM on April 2, 2010 [2 favorites]
I think clay tablets are underrated. The feel of tough, rigid blocks of clay under your fingers beats out that flimsy, wussy, flippy-flappy newfangled paper any day. My Sumerian priest-neighbour swears by them, too.
posted by WalterMitty at 7:51 PM on April 2, 2010
posted by WalterMitty at 7:51 PM on April 2, 2010
See, the thing is, I work in assistive technology: providing alternative mechanisms to the standard keyboard/mouse interface for blind people, Braille users, people who have no arms, people who can't stop their hands shaking - that sort of thing. This requires mechanisms that, yes, identify control structures and permit different controls and approaches.
If this stuff works for that purpose, that's awesome.
posted by The Devil Tesla at 8:24 PM on April 2, 2010
If this stuff works for that purpose, that's awesome.
posted by The Devil Tesla at 8:24 PM on April 2, 2010
You should try an OS that allows focus-follows-mouse
No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
I just wish Windows would realize that the window I'm hovering over should always receive all scroll wheel events, as X figured out long ago. Surely you don't object to that, at least.
posted by Xezlec at 8:57 PM on April 2, 2010 [1 favorite]
No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
I just wish Windows would realize that the window I'm hovering over should always receive all scroll wheel events, as X figured out long ago. Surely you don't object to that, at least.
posted by Xezlec at 8:57 PM on April 2, 2010 [1 favorite]
"I just wish Windows would realize that the window I'm hovering over should always receive all scroll wheel events..."
Is this Windows per se , or a API not fully implemented somehow? I'm thinking about how MS Access, even 2007, seems to be very erratic when responding to scroll wheel events.
posted by sneebler at 9:56 PM on April 2, 2010
Is this Windows per se , or a API not fully implemented somehow? I'm thinking about how MS Access, even 2007, seems to be very erratic when responding to scroll wheel events.
posted by sneebler at 9:56 PM on April 2, 2010
You should try an OS that allows focus-follows-mouse
No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
I just wish Windows would realize that the window I'm hovering over should always receive all scroll wheel events, as X figured out long ago. Surely you don't object to that, at least.
Scroll wheel -> yes. But seconding GuyZero, focus definitely not. I had one second-year CS course where the lab machines had this, and being my first exposure it horribly skewed my (and my entire class's) perspective of Linux for several years. I still occasionally have to clarify to classmates, when they find out I use Ubuntu, that no most *nix systems are not that bad.
One thing that drove me nuts about it was you could be peacefully coding when your arm accidentally brushes your mouse. Oops - you just entered a dozen keystrokes in the middle of another program, and holy shit did that screw everything up.
posted by Arandia at 4:33 AM on April 3, 2010
No, you should not. I hate it with the passion of a thousand burning suns. It's really annoying. No one's used it since mid-90's X window managers for a reason.
I just wish Windows would realize that the window I'm hovering over should always receive all scroll wheel events, as X figured out long ago. Surely you don't object to that, at least.
Scroll wheel -> yes. But seconding GuyZero, focus definitely not. I had one second-year CS course where the lab machines had this, and being my first exposure it horribly skewed my (and my entire class's) perspective of Linux for several years. I still occasionally have to clarify to classmates, when they find out I use Ubuntu, that no most *nix systems are not that bad.
One thing that drove me nuts about it was you could be peacefully coding when your arm accidentally brushes your mouse. Oops - you just entered a dozen keystrokes in the middle of another program, and holy shit did that screw everything up.
posted by Arandia at 4:33 AM on April 3, 2010
I've seen a research poster from one of the creators, and talked to them for 20 minutes or so. In a research context, the project is interesting as a way to test HCI ideas on real-world programs. There is always some doubt as to the usefulness of new ideas in HCI/programming languages/other CS fields; steps towards seeing immediately how these ideas fare in real situations is extremely useful. Of course there are things to improve, but the fact that they have no shortage of feedback means a few people have easy dissertation (and conference paper) topics.
posted by mezamashii at 11:22 AM on April 3, 2010
posted by mezamashii at 11:22 AM on April 3, 2010
Machine vision to find the widgets on-screen seems like super-overkill. Seriously, how expensive is that? What's the false positive rate? If I draw a rectangle on the screen, is it going to be identified as a button?
The premise is "we don't have the source code so we need to do this", but you could do most of this by wrapping a shim around the GUI toolkit's shared library, and it would be a lot faster.
posted by qxntpqbbbqxl at 4:30 PM on April 3, 2010
The premise is "we don't have the source code so we need to do this", but you could do most of this by wrapping a shim around the GUI toolkit's shared library, and it would be a lot faster.
posted by qxntpqbbbqxl at 4:30 PM on April 3, 2010
My previous post comes off as really negative. I should also add that this is totally cool.
posted by qxntpqbbbqxl at 4:33 PM on April 3, 2010
posted by qxntpqbbbqxl at 4:33 PM on April 3, 2010
you could do most of this by wrapping a shim around the GUI toolkit's shared library
The reason this is cool is because it works in the cases where you can't do that.
posted by grouse at 5:31 PM on April 3, 2010
The reason this is cool is because it works in the cases where you can't do that.
posted by grouse at 5:31 PM on April 3, 2010
If this would make scroll bars in Flash actually work like scroll bars - respond to mouse wheel and not force me to click and drag - let's implement the hell out of it already.
posted by caution live frogs at 11:53 AM on April 5, 2010 [1 favorite]
posted by caution live frogs at 11:53 AM on April 5, 2010 [1 favorite]
« Older The Lifesaver bottle. It does what it says. | Game, set, match. Newer »
This thread has been archived and is closed to new comments
posted by ArgentCorvid at 7:49 AM on April 2, 2010