The free dataset in the maw of FAANG
November 19, 2020 3:38 PM   Subscribe

OpenStreetMap (OSM) is now at the center of an unholy alliance of the world’s largest and wealthiest technology companies. The most valuable companies in the world are treating OSM as critical infrastructure for some of the most-used software ever written.

The four companies in the inner circle— Facebook, Apple, Amazon, and Microsoft— have a combined market capitalization of over six trillion dollars. In almost every other setting, they are mortal enemies fighting expensive digital wars of attrition. Yet they now find themselves eagerly investing in and collaborating on OSM at an unprecedented scale (more on the scale later).

Possibly the best lede on a Medium post in 2020 (yes, single-link Medium), here's a story about corporate contributions to a crowd-sourced dataset coming together to improve it.
posted by k3ninho (33 comments total) 26 users marked this as a favorite
 
I'm a long-time craft mapper, so I have feelings about this. The Corps are only interested in streamlining data that could be useful for deliveries: every single access road has been meticulously mapped by them. Addresses? Restaurants? Bus stops? None at all.
posted by scruss at 4:00 PM on November 19, 2020 [9 favorites]


Oh man I wish I could meaningfully participate in this discussion without breaking workplace rules, so I’ll just say that I’m happy to still see the term “craft mapper” in use four years on.
posted by migurski at 5:07 PM on November 19, 2020 [4 favorites]


I appreciated the article for highlighting an important phenomenon. Also a sort of back door cultural shift at OSM; I'm glad that these institutions have figured out how to contribute despite some very unpleasant opposition. OSM data is really amazing these days.

I had a chance to do some map-making for Highlands Papua New Guinea recently. For this area Google Maps sort of has the main highway on it but nothing else. OSM has whole road networks for villages, all the dirt roads, a whole lot of detail. I'm not sure if it was placed there by commercial interests or humanitarian ones, but either way it's a win for everyone.

scruss, nothing's stopping you from continuing craft mapping. You can put the addresses and restaurants and bus stops on that access road. They've saved you the trouble of tracing the line string from your bicycle's GPS logs.
posted by Nelson at 5:30 PM on November 19, 2020 [7 favorites]


The post talks about the involvement of corporations with OSM, but doesn't really touch on the direction of the involvement.

It's no where close to my department, so I don't have any insider info, but it's public knowledge that my employer (everyone's favorite fruit company) has been rolling out some extreme updates to our mapping stuff over the last 2-3 years, so it's not a surprise to me we're (according to this post anyway) pushing lots of changes upstream. End users can report problems via the Maps app, so perhaps we are being good citizens by passing on the fix to wherever the incorrect source info came from, or perhaps we just fix everything everywhere, I don't know.

I can say, as a private citizen that has no knowledge of how our maps data get slurped and processed (besides our white vans I've seen driving around on public streets), that all the relevant parties (Google, Apple, USPS, UPS, FedEx, etc) get and use all sorts of different data and they all use it differently.

Here's an example:

Last December, I bought a house where everyone on the planet, including the United States Postal Service, said the house was on a "Avenue". However, my title search said it was actually a "Drive". I drove over one night and I found out that, yes, I was buying a house on a Drive no matter what literally everything including the last three property transfers recorded at the county said.

Well, this caused all kinds of problems because the banks, insurance company, the util companies, the digital records for my new city, the digital records of the county, all said "Ave". Only the old scanned parcel maps and the physical sign on the street itself said "Dr." All kinds of higher ups had to sign off because the bank was worried the insurance co might not pay out if my house burned down, and insurance co could only shove the Dr. change partway through their system before some "this address does not exist" check rejected the printing of the docs. Anyway, it made my house purchase kinda nerve-y, but it all worked out.

On the day I got the keys, I submitted changes to Apple via our customer facing app, to Google via their site, and I logged into OSM and changed it there. A month later I checked everywhere and found...that exactly zero systems (besides OSM) had picked up the change. Even OSM would show the bad info on certain views (I'm not very good at making changes there apparently). Turns out the USPS's Coding Accuracy Support System (CASS) said I was an Ave, and if they said that, no one was going to contradict them.

A month later after that I happed to switch to a local insurance agent, and in an aside I mentioned my address debacle. He said he had a contact in my city government, and two days latter my address was updated in CASS, and a week after that both Apple Maps and the USPS address verification website thing were correct. UPS was fixed a couple weeks after that, and at some point FedEx followed.

Google, 11 months later, has not updated. Either has Postmates. I used a junk hauling company, and it took an extra week to get setup because their system keep deleting my order because I was putting my correct address and their system said "hahahaha yeah right" and kept deleting my work order.

Anyway, the TL;DR; of my sad address struggle is: Everybody has very many sources of information, and even the companies named in this post are not just using OSM data verbatim in their products.
posted by sideshow at 5:34 PM on November 19, 2020 [43 favorites]


sideshow,

regarding your osm showing the outdated 'bad' info on certain views, it's that osm's maps (on openstreetmap.org) have a lag time of implementing your changes anywhere between a few hours for some zoom levels on the default map, to several weeks on the other layers.
posted by fizzix at 5:38 PM on November 19, 2020 [3 favorites]


This is a fascinating article & phenomenon, thank you for posting it.

The theme here is that there's an uneasy coincidence of interests between amateur geographers and corporations, which I'm sure reflects part of what's going on, but it seems to me the longest-standing and enduring interested parties in having a reliable dataset of the world—militaries and intelligence agencies, who've been doing the job for hundreds if not thousands of years—are still silent partners here, and still retain control of a lot of the key technologies, from GPS/GLONASS to geospatial standards. If they weren't already involved in OSM (openly or otherwise), I am certain that they're involved in the corporates' use of it.

Take your pick, have a collision of interests with Apple and Google, or a collision of interests with the kind of people who plan warhead delivery, anywhere on earth.
posted by Fiasco da Gama at 5:42 PM on November 19, 2020 [1 favorite]


(I should clarify: your changes are immediately included into the OpenStreetMap database as soon as you upload your changes; but the maps may not immediately display those changes).
posted by fizzix at 5:44 PM on November 19, 2020 [1 favorite]



From a perspective whose relationship at this point consists of editing a couple times a week to make some small updates to my locality:

Joe does an excellent job of giving a concisely summarizing the changes in their involvement of OSM over the past 3-4 years. I'm mostly ambivalent about it.

While editing, I come across errors that the paid mappers make a few times a month in my area
(have no idea on the percentage of errors, if I had to guess, purely out of my air, it's 5%.
They draw roads (and buildings) using outdated aerial imagery that no longer exist or their addition or modify roads that break public transportation routes. It's mildly annoying, enough to sometimes leave a comment on the changeset and then revert (which undoes) their changes; but not enough to keep a long list of their breaking list nor do I have the time nor care to review every single commit that they make in my area.
posted by fizzix at 6:12 PM on November 19, 2020 [3 favorites]


Last December, I bought a house where everyone on the planet, including the United States Postal Service, said the house was on a "Avenue". However, my title search said it was actually a "Drive".

For a second, I thought you might be living on my road, but mine is an "Drive" that everyone thinks is a "Street".
I basically solve it by leaving off the descriptor, so "123 Elm" rather than "123 Elm St." because there is only one "Elm" in my zip code.

That way whatever system someone is using can just fill in their preferred ending and everyone is happy.
posted by madajb at 7:42 PM on November 19, 2020 [1 favorite]


Sounds like a good thing, but not knowing anything about it, my thought is, what are they up to?
posted by blue shadows at 8:15 PM on November 19, 2020


He said he had a contact in my city government, and two days latter my address was updated in CASS

Guess it's faster to change the name of the street than to have a correct street sign delivered and installed...
posted by save alive nothing that breatheth at 8:30 PM on November 19, 2020 [1 favorite]


Two days ago, I had to visit a doctor's office. I checked the location in Apple Maps, and was suspicious—the location seemed wrong based on my general knowledge of the area. Indeed it was: the correct location was on the opposite side of the road. I submitted an update to Apple Maps, and today received a notification that it had been accepted. I was kind of surprised they closed that loop. But it's still showing in the wrong place. I've submitted updates to Google Maps before, one of which "took," one didn't.

One thing I'm really curious about is transitory map changes. There's been a massive highway project in my area, going on for years. It's disrupted all of the roads nearby, and if you're going anywhere around there, the way you get from point A to point B can change from one week to the next. I've been amazed that Google Maps' directions have kept pace with this, guiding me through temporary workarounds correctly (for those of you in Austin, I'm talking about the weird junction between US 183 and Airport Blvd).
posted by adamrice at 8:47 PM on November 19, 2020


OSM acolytes hate this comparison [to Wikipedia] in the much same way baseball players resent when people describe the sport as “cricket for fat people.” While vaguely truthful, it doesn’t quite get to the spirit of the thing.
Also, cricket for fat people is called cricket.
posted by flabdablet at 8:58 PM on November 19, 2020 [2 favorites]


maybe it's time for cabinet level position -- say, a department of technology, like alongside the department of transportation, energy, labor, treasury, etc. -- to oversee some of this stuff?
posted by kliuless at 10:10 PM on November 19, 2020


I basically solve it by leaving off the descriptor, so "123 Elm" rather than "123 Elm St." because there is only one "Elm" in my zip code.

That wouldn't work too well in parts of England. (Example: search Nowell Street, Leeds.)
posted by Cardinal Fang at 11:24 PM on November 19, 2020 [2 favorites]


In London, I've seen a lot of clusters of a street name replicated in various types of roads, as well as squares and such. Some of that probably has to do with historical landowners naming thoroughfares on their land after other land they owned/titles they held or similar.
posted by acb at 12:54 AM on November 20, 2020


sideshow, I find it really shocking that all these institutions rely on the street address. I thought real estate used parcel numbers specifically to avoid these issues (and also the issue where you're so far out in the sticks that there are no addresses)
posted by ymgve at 1:24 AM on November 20, 2020


> I've been amazed that Google Maps' directions have kept pace with this, guiding me through temporary workarounds correctly

They have thousands of Android GPS trackers in cars feeding them the changes in realtime.
posted by goinWhereTheClimateSuitsMyClothes at 2:42 AM on November 20, 2020 [2 favorites]


I'm a long-time craft mapper, so I have feelings about this. The Corps are only interested in streamlining data that could be useful for deliveries: every single access road has been meticulously mapped by them.

You weren't kidding about every single road -- I looked up the farm that's been in my family for three generations and OSM includes not only the private access road that runs through the farmstead but also a trail that's nothing more than a pair of tire ruts that leads to a ford across the creek. How exactly are they collecting this data?
posted by nathan_teske at 7:50 AM on November 20, 2020


I reported a map error to Apple for literally years before they fixed it. I asked for directions to a small town in northern MI. They provided directions to an intersection in southern MI. It wasn't even in the same peninsula. To get accurate driving directions I had to plug in the ZIP code, not the town name. Because apparently the town name - which was used everywhere - was not as important as the name of the intersection (which was NOT a town, but was in a small town with an entirely different name).

The last time I reported it, I am pretty sure I included a screenshot of the Apple map with the incorrect information, along with a screenshot of Google Maps (with the right info), OSM (with the right info), Bing maps (with the right info), fuckin' MapQuest (with the right info), and maybe even the goddamn Wikipedia entry too.

Apple Maps is now correct.

(In contrast, a problem report to Google Maps indicating that they had mis-named a local business [typo in the name]? I included a screenshot of Google's streetview of the building with the name correct on the sign, they fixed it within days.)

Mapping is hard I guess.
posted by caution live frogs at 8:26 AM on November 20, 2020 [1 favorite]


Amazon Logistics is using high-resolution satellite imagery to add their edits, and Facebook is doing some kind of AI labeling. I'm guessing Amazon is going to use it to improve ground deliveries, and probably to build out drone delivery one of these days.

BTW you can snoop on map contributors from a number of perspectives here.
posted by RobotVoodooPower at 8:54 AM on November 20, 2020 [2 favorites]


nathan_teske:

How is the data collected (this is from an American perspective...OSM's coverage/usage varies greatly across the world, even within some countries) :

It depends on the location and the mapper there's several ways, I'll list them in decreasing popularity:

- A user manually traces over a feature (building/road, river) using their mouse over aerial imagery taken by satellites or airplanes Corporations competing with google maps (yahoo in the late 00s; more recently maxar and bing), have provided aerial imagery since in the late 00s, ; some governments publicly release aerial imagery [collected for agricultural, land conservation, surveying/urban planning] for free).
(Drones are also increasingly used but that method of collection aerial imagery for the purposes of OSM but is mostly done on a very small scale, like a less than a town's size by NGOs [local and international] and individual hobbyists; none of the corporations are using drones yet).

Amazon will pay people (iirc, based in India) to use the available aerial imagery sources (the same ones available to other OSM editors even you and me) to trace the ways. Where do they trace? As far as I know, that's based on GPS recordings provided by the amazon prime drivers.

- the data (roads, buildings) are collected by a governmental entity and then mass imported into OSM by users. This is extremely controversial within the OSM community (you could write a theses why) but the most of the road network in the USA was first added to OSM this way.

- GPS traces from hobbyists that use their smart phones and/or specific apps to record their location and make little voice notes of particular details that they encounter (this is being used less and less because the accuracy of GPS traces (provided by the devices) haven't improved in the past 10-15 years and aerial imagery is becoming clearer, available, accurate, and up to date).

- Some of the competing corporations (strava) have provided layers of semi-anonymized gps traces for users to trace over (don't think the strava is available anymore..)

- mapillary (a crowd-sourced google-street-view; a swedish startup, bought by facebook earlier this year) providing photographs that users could use and reference to then draw the objects manually.

- Facebook, with its rapID program, uses machine learning based on aerial imagery of existing roads that would detect the ruts of the road.
posted by fizzix at 9:11 AM on November 20, 2020 [4 favorites]


You can use apps like Go Map! to add little details to your surroundings, things as mundane as park benches and water fountains and boardwalks, or annotated signs and art installations.

I used to think the majors could also use it as a collaborative directory of retail businesses, but that aspect is sparsely populated (at least in the US) and Google/Facebook/Yelp have their own incentives to silo their data.
posted by RobotVoodooPower at 9:25 AM on November 20, 2020


sideshow, I find it really shocking that all these institutions rely on the street address. I thought real estate used parcel numbers specifically to avoid these issues (and also the issue where you're so far out in the sticks that there are no addresses)

Yeah, it works this way at the "where the cold and mechanical legal rubber meets the road" level. Unfortunately, with buying a home, there are levels upon levels that just rely on humans and whatever feelings they might have at that moment.

The problem is that buying a house is a race against the clock, and someone times you have to just humor a power-hungry asshole* because the closing date is very rapidly approaching, and if you fail, you not only lose the house but $50k in earnest money. I literally got final approval 36 hours before escrow closing (about 14 days after I was supposed to remove all my contingencies, but had not).

* The address situation wasn't even the worst thing that happened with this guy. TCF (the lender in question) wanted some document from the IRS by noon the next day, and I had to call the IRS appointment line and say "find me an appointment at an office anywhere in the US where my plane flight and subsequent commute gets me there by 11am PST". Fortunately, I was able to get one at an office I could drive overnight to. But, the IRS dude said what TCF wanted didn't exist, but I convinced him to literally pull up a blank form and just filled out what TCF wanted, and then give it his official IRS stamp. I left the IRS office to find a VM from my agent that said the lender had already sent the docs to escrow BEFORE I walked into the appointment, and that my 10 hour roundtrip that included an overnight stay and my 30 mins of social engineering the IRS agent was totally useless gnar gnar gnar!
posted by sideshow at 10:26 AM on November 20, 2020


That wouldn't work too well in parts of England. (Example: search Nowell Street, Leeds.)

Nowell Street
Nowell Approach
Nowell Lane
Nowell Walk
Nowell Parade
Nowell Avenue
Nowell Grove
Nowell Terrace
Back Nowell Terrace
Nowell Place
Back Nowell Place
Nowell Mount
Back Nowell Mount
Nowell End Row
Nowell Crescent
Back Nowell Crescent
Nowell View
Nowell Court
posted by yonega at 12:48 PM on November 20, 2020


Well, that ended … Nowell.

I used the term craft mapper ironically. I'm only slightly gate-keepery, but not like the guy who stopped an entire community and government-supported building import for all of Canada because he hadn't been personally consulted about his neighbourhood edits.
posted by scruss at 1:11 PM on November 20, 2020 [3 favorites]


Also witness Sage Hen Dr, which has Sage Hen cir/ct/cv hanging off it. Houses on Sage Hen Dr at least has different address numbers, but houses on the other streets have the same numbers.
posted by adamrice at 1:15 PM on November 20, 2020


Same thing happens in many modern subdivisions in Canada. They'll all have the same street name varying only the street type. Some mix of:

Foo Street
Foo Lane
Foo Avenue
Foo Boulevard
Foo Crossroad
Foo Trail
Foo Loop
Foo Place
Foo Wynd
Foo Crescent
Foo Gate
Foo Bend
Foo Circle
Foo Close
Foo Walk (sometimes ironically with no sidewalks on the street)
Foo Court
Foo Landing
Foo Terrace

Naturally the development will be identified as Foo. Like adamrice said good developers don't re-use numbers on different streets, others start all streets at 1 or 100. Back when I was doing service work I would have liked to punch those guys.

Worst place I'd ever lived though was on West Road. That's it, the Name of the Road was West. Practically impossible to get deliveries and hard to search for now.
posted by Mitheral at 1:27 PM on November 20, 2020


I used to live near an odd intersection--Blacklidge Drive & Blacklidge Drive

Google and OSM both show it as Blacklidge Drive & Blackridge Drive. Apple maps just doesn't display any name for "Blackridge Drive. The actual street signs indicate that the north/south street and the east/west street are both East Blacklidge Drive. Go figure.

I'm glad that I lived a few blocks away so I didn't have to deal with tons of confused delivery people.
posted by threecolorable at 2:49 PM on November 20, 2020


Some of the competing corporations (strava) have provided layers of semi-anonymized gps traces for users to trace over (don't think the strava is available anymore..)

With a paid Strava account you can use the high-res heatmap for tracing. I wrote up a how-to on it here.

I use this frequently for mapping hiking/XC skiing/mountain bike trails since they are generally not visible in aerial imagery, and the path-consensus from the heatmaps is better than pretty much anything besides getting a multi-thousand-dollar GNSS receiver and hand-surveying them myself.

I just wish there was a tool which'd automatically find the center of heatmap'd routes and let me turn that into ways. That'd be super duper helpful.
posted by c0nsumer at 7:20 PM on November 20, 2020 [1 favorite]


That wouldn't work too well in parts of England. (Example: search Nowell Street, Leeds.)

That is excellent.
posted by madajb at 10:59 PM on November 20, 2020


That wouldn't work too well in parts of England. (Example: search Nowell Street, Leeds.)

Thank god for the way we use postcodes. Averages out at 15 properties per postcode, and at that point the Postie will generally figure it out.

One of my friends likes getting mail with essentially puzzle clues for the postal service. I once sent him a postcard with his name, that he lived with someone called Kelly, that he got weirdly addressed letters, and what turned out to accidently be the next town over.

Turned up in less than a week.
posted by MattWPBS at 6:42 AM on November 21, 2020 [2 favorites]


If there's one thing Britain genuinely leads the world in, it'd be its postcode system. It is a marvel of good design and systems thinking of a sort that's sadly not common there.
posted by acb at 6:51 AM on November 21, 2020


« Older How the U.S. Military Buys Location Data from...   |   A 32-year-old man marrying a barely 20-year-old... Newer »


This thread has been archived and is closed to new comments