Let Freedom Ring...and Be Mapped
July 1, 2014 1:37 PM Subscribe
Esri Enables Federal Agencies To Open GIS Mapping Data To The Public ESRI is the world's leading maker of GIS software. Their initiative is incredibly important in making mapped/mappable data available to the world. They are basically giving government agencies an Easy Button for opening this up to the public.
GIS (Geographic Information System) is kind of like SimCity for the real world: It is basically a database with a mappable component. But, to be a true GIS, it needs to be tied into some larger data gathering system and often is for government agencies like urban planning offices. Most data can be tied to a physical location (like a street address or lat-long coordinates) and certain types of "normal" databases (such as Excel) are readily compatible and importable to some GIS software packages. So, if you have data and the right software, you can powerfully visualize that data in a map.
Here is a relatively recent court ruling related to the above: GIS mapping data is public, not subject to software fee, California high court rules
Why does all of this matter? Because a picture is worth a thousand words and GIS mapping technology is helping underprivileged communities get better services — from education and transportation to health care and law enforcement — by showing exactly what discrimination looks like.
GIS (Geographic Information System) is kind of like SimCity for the real world: It is basically a database with a mappable component. But, to be a true GIS, it needs to be tied into some larger data gathering system and often is for government agencies like urban planning offices. Most data can be tied to a physical location (like a street address or lat-long coordinates) and certain types of "normal" databases (such as Excel) are readily compatible and importable to some GIS software packages. So, if you have data and the right software, you can powerfully visualize that data in a map.
Here is a relatively recent court ruling related to the above: GIS mapping data is public, not subject to software fee, California high court rules
Why does all of this matter? Because a picture is worth a thousand words and GIS mapping technology is helping underprivileged communities get better services — from education and transportation to health care and law enforcement — by showing exactly what discrimination looks like.
yet, ESRI is like Adobe (photoshop, flash, desinger, etc) or Autodesk (CAD software), they have such a stranglehold on the industry and format. Yes, I know there are gnu/free alternatives out there, but they are all pretty poor in comparison.
posted by k5.user at 2:25 PM on July 1, 2014 [5 favorites]
posted by k5.user at 2:25 PM on July 1, 2014 [5 favorites]
Great that the data is being made more available - it's a shame that ESRI's flagship software is both too expensive for anyone but businesses/consultants/universities to run, and buggy as all hell. 2014 and you still can't convert a floating-point vector to a raster. Or run it on anything but Windows.
Thankfully, the free, open source Quantum GiS is catching up quickly, or you can play with the data in web-based GIS databases like CartoDB or Mapbox.
posted by Jimbob at 2:26 PM on July 1, 2014 [11 favorites]
Thankfully, the free, open source Quantum GiS is catching up quickly, or you can play with the data in web-based GIS databases like CartoDB or Mapbox.
posted by Jimbob at 2:26 PM on July 1, 2014 [11 favorites]
This is really neat. I'm playing with the map at healthycity.org (linked to in the last article of the FPP).
posted by insectosaurus at 2:29 PM on July 1, 2014 [1 favorite]
posted by insectosaurus at 2:29 PM on July 1, 2014 [1 favorite]
Where can a person find background on the Sierra Club / Orange County argument? I guess what I'm looking for is some discussion of why Orange County were being tools in the first place -- thus far I've mostly found poor-quality coverage that doesn't really explain much on their side of the case.
I'm trying to assume they're not just jerks. I mean, they could just be jerks.
posted by aramaic at 2:29 PM on July 1, 2014
I'm trying to assume they're not just jerks. I mean, they could just be jerks.
posted by aramaic at 2:29 PM on July 1, 2014
yet, ESRI is like Adobe (photoshop, flash, desinger, etc) or Autodesk (CAD software), they have such a stranglehold on the industry and format.
I use and enjoy GIS, and I so wish there was a viable alternative. Arc is buggy, finicky, and not very user friendly, but as a tool for visualization and analysis GIS is incredibly powerful.
The other barrier is the size of data sets. Even with a fast connection and a new computer things can be too big to efficiently send back and forth, and impossible if you are in a place with limited connection speeds or old equipment you are hosed.
posted by Dip Flash at 2:40 PM on July 1, 2014
I use and enjoy GIS, and I so wish there was a viable alternative. Arc is buggy, finicky, and not very user friendly, but as a tool for visualization and analysis GIS is incredibly powerful.
The other barrier is the size of data sets. Even with a fast connection and a new computer things can be too big to efficiently send back and forth, and impossible if you are in a place with limited connection speeds or old equipment you are hosed.
posted by Dip Flash at 2:40 PM on July 1, 2014
Oh, this could be really great. If this results in state/local governments making their data more accessible to the public, that's huge. There are treasure troves of data out there that are just a pain in the ass to obtain and keep up to date. (I use GIS daily for engineering analysis and mapping flood risk.)
As much as I hate to see Esri's monopoly grow... I really have trouble not being excited by the possibilities here.
posted by whitecedar at 2:42 PM on July 1, 2014
As much as I hate to see Esri's monopoly grow... I really have trouble not being excited by the possibilities here.
posted by whitecedar at 2:42 PM on July 1, 2014
aramaic, I don't know where you can get more info on that case. But I will suggest that maybe one reason they were trying to charge a fee is because 60% of the expense for a GIS is good data. The right people are the most critical component for making it work but data is the single most expensive part of it. (There are five components to a complete GIS. In addition to data and the people running the system, there is hardware, software, and procedures.)
posted by Michele in California at 2:43 PM on July 1, 2014
posted by Michele in California at 2:43 PM on July 1, 2014
ESRI Blue? The article reads like a press release...
Out of all of the software monopolies, ESRI's might be one of the most harmful. GIS is critically important to the further development of our civilization, and ESRI has had a knack for stomping out competition and proliferating proprietary formats.
I don't trust them, and this development doesn't help. They're providing "free" services that will encourage further lock-in.
posted by schmod at 2:57 PM on July 1, 2014 [14 favorites]
Out of all of the software monopolies, ESRI's might be one of the most harmful. GIS is critically important to the further development of our civilization, and ESRI has had a knack for stomping out competition and proliferating proprietary formats.
I don't trust them, and this development doesn't help. They're providing "free" services that will encourage further lock-in.
posted by schmod at 2:57 PM on July 1, 2014 [14 favorites]
Yeah, I did some work with QGIS too but obviously ESRI has cornered most of the market. Hopefully some of these decisions about making data open to the public will be an incentive to improve open source GIS programs.
posted by mlle valentine at 3:01 PM on July 1, 2014
posted by mlle valentine at 3:01 PM on July 1, 2014
The thing about ArcGIS is that it includes a very large set of analytical tools that work pretty well, and organizations that use it tend to buy in all the way (it's called the Microsoft of GIS for a reason).
If I'm not mistaken, all ESRI is doing here is allowing people to publish data publicly using standard formats, without charge -- something that open source solutions have made trivial for a long time (MapServer/GeoServer). And they're doing that, in part, because the court has decided that public geodata can't be under a software license.
What this won't change is the fact that regardless of what infrastructure you use, you still need designers and programmers to build public-facing web applications (unless you're just going to set up a public-facing WMS/WFS/etc. and let people use the data however they choose). And governments will still have to pay huge licensing fees to ESRI to use their tools (not that open source GIS doesn't come with significant costs too).
It looks like, if anything, ESRI is just catching up.
posted by klanawa at 3:07 PM on July 1, 2014 [3 favorites]
If I'm not mistaken, all ESRI is doing here is allowing people to publish data publicly using standard formats, without charge -- something that open source solutions have made trivial for a long time (MapServer/GeoServer). And they're doing that, in part, because the court has decided that public geodata can't be under a software license.
What this won't change is the fact that regardless of what infrastructure you use, you still need designers and programmers to build public-facing web applications (unless you're just going to set up a public-facing WMS/WFS/etc. and let people use the data however they choose). And governments will still have to pay huge licensing fees to ESRI to use their tools (not that open source GIS doesn't come with significant costs too).
It looks like, if anything, ESRI is just catching up.
posted by klanawa at 3:07 PM on July 1, 2014 [3 favorites]
Not to be a jerk, but this is really a thin post that is mostly based on a puff piece about ESRI and how they will solve all of our problems- something that they have been saying for years. No mention of the Open Geospatial Consortium standards body, which is where huge amounts of innovation are happening right now. I work really hard with a really dedicated group of folks to make federal data available (you can see an example in my projects posts) and it is through the open source and open standards communities that we have been able to move forward and make really good products that others (not just those that can afford ESRI licensing fees) can use.
I don't understand how they can say that things are open when for things to work well on their platform, everyone involved has to be using the same stack. Also, you never know when ESRI will decide to drop support for something, leaving you stranded.
The federal government is moving in a number of directions at once, but things really do seem to be focusing on data.gov, which is moving to an open-source, standards-compliant cataloging tool called CKAN. Really cool things are happening in the federal open science data space, and it is not because of the tool in the post.
The open source geospatial community is strong and growing stronger, because many organizations just can't afford to keep using and paying for ESRI products anymore. A city can drop ESRI licensing and instead use that money to contribute to core development of open source tools for all to use. PostGIS is incredibly powerful geospatial database, especially when you put Geoserver on top of it and openlayers or leaflet in front of the Geoserver Web Map Services. QGIS is also growing into a solid product, though I have the least experience with it. A good place to keep track of cool things in the open geospatial community is through OSGeo.
Now back to planning how to make a Web Feature Service for a few million sampling sites to make the data collected at those sites more visible and accessible.
posted by rockindata at 3:10 PM on July 1, 2014 [35 favorites]
I don't understand how they can say that things are open when for things to work well on their platform, everyone involved has to be using the same stack. Also, you never know when ESRI will decide to drop support for something, leaving you stranded.
The federal government is moving in a number of directions at once, but things really do seem to be focusing on data.gov, which is moving to an open-source, standards-compliant cataloging tool called CKAN. Really cool things are happening in the federal open science data space, and it is not because of the tool in the post.
The open source geospatial community is strong and growing stronger, because many organizations just can't afford to keep using and paying for ESRI products anymore. A city can drop ESRI licensing and instead use that money to contribute to core development of open source tools for all to use. PostGIS is incredibly powerful geospatial database, especially when you put Geoserver on top of it and openlayers or leaflet in front of the Geoserver Web Map Services. QGIS is also growing into a solid product, though I have the least experience with it. A good place to keep track of cool things in the open geospatial community is through OSGeo.
Now back to planning how to make a Web Feature Service for a few million sampling sites to make the data collected at those sites more visible and accessible.
posted by rockindata at 3:10 PM on July 1, 2014 [35 favorites]
If you are wondering what is making federal data more accessible, it is policy, coming from the top. For example, this memo about making research data open and accessible from the White House Office of Science and Technology Policy, and this Executive Order making open and machine readable the default for government data.
There is also the fact that there are a lot of people that really really care about this stuff, and there are also finally the tools and standards to make it possible to not only make the data available, but in a form that is usable, discoverable, and consumable by many different stakeholders.
posted by rockindata at 3:25 PM on July 1, 2014 [3 favorites]
There is also the fact that there are a lot of people that really really care about this stuff, and there are also finally the tools and standards to make it possible to not only make the data available, but in a form that is usable, discoverable, and consumable by many different stakeholders.
posted by rockindata at 3:25 PM on July 1, 2014 [3 favorites]
rockindata, thanks for all the great links.
However, the post was sort of intentionally "thin" as it was in response to a request by a mefite who explicitly said she doesn't know much about GIS and would be interested in hearing from someone knowledgeable on the subject. A quick search of previous GIS posts suggests they tend to not get much comment. My experience suggests this is likely because it is a dense topic and hard to approach for outsiders. So this post was intended as a GIS 101 primer, in hopes of introducing some of the basic concepts in an approachable manner and tying them to something with some depth and meaning for lay people who have not had training, don't understand what all the terms are, and don't understand how this is relevant to their lives (but are, nonetheless, intelligent and interested in the topic).
When I was in GIS school and when I was active in an urban planning forum, one of the things insiders complained a great deal about was inability to explain what they did to outsiders who had no idea what GIS (or even urban planning) really was. Any time I talked about "it's like SimCity for the real world" that really hit a nerve for people. It had conceptual depth in a nutshell.
So that explanation is time tested among a variety of audiences as approachable and a means to help bridge the gap between very knowledgeable insiders like yourself and lay people who would like to know more but are having some trouble figuring out where to start.
posted by Michele in California at 3:34 PM on July 1, 2014 [3 favorites]
However, the post was sort of intentionally "thin" as it was in response to a request by a mefite who explicitly said she doesn't know much about GIS and would be interested in hearing from someone knowledgeable on the subject. A quick search of previous GIS posts suggests they tend to not get much comment. My experience suggests this is likely because it is a dense topic and hard to approach for outsiders. So this post was intended as a GIS 101 primer, in hopes of introducing some of the basic concepts in an approachable manner and tying them to something with some depth and meaning for lay people who have not had training, don't understand what all the terms are, and don't understand how this is relevant to their lives (but are, nonetheless, intelligent and interested in the topic).
When I was in GIS school and when I was active in an urban planning forum, one of the things insiders complained a great deal about was inability to explain what they did to outsiders who had no idea what GIS (or even urban planning) really was. Any time I talked about "it's like SimCity for the real world" that really hit a nerve for people. It had conceptual depth in a nutshell.
So that explanation is time tested among a variety of audiences as approachable and a means to help bridge the gap between very knowledgeable insiders like yourself and lay people who would like to know more but are having some trouble figuring out where to start.
posted by Michele in California at 3:34 PM on July 1, 2014 [3 favorites]
I work with the Government's GIS data nearly every day at my company, along with GIS data licensed from private sources. We specialize in aggregating data and building visualizations, our bread and butter being very flexible maps.
One of my jobs is to build custom geospatial shapes and project all kinds of data into novel 2-D shapes on the earth's surface. I use free software tools to work with these shapes, namely Python packages like Shapely and Fiona. I've only worked with ESRI as far as building a parser to reinterpret .shp and .dbf files into a format that I could save to a geospatial database, so it is possible to take advantage of this great abundance without dealing with clunky software. Still, I am thankful to ESRI for building the software needed to make these rich and fantastic data sources.
I'm even giving a talk about how to work with 2-D data and geospatial data at PyOhio next month! I could talk at length about different granularities of shape data and their quality, but I'll just say instead it is a really exciting time to be in the geospatial data business.
posted by Alison at 3:42 PM on July 1, 2014 [6 favorites]
One of my jobs is to build custom geospatial shapes and project all kinds of data into novel 2-D shapes on the earth's surface. I use free software tools to work with these shapes, namely Python packages like Shapely and Fiona. I've only worked with ESRI as far as building a parser to reinterpret .shp and .dbf files into a format that I could save to a geospatial database, so it is possible to take advantage of this great abundance without dealing with clunky software. Still, I am thankful to ESRI for building the software needed to make these rich and fantastic data sources.
I'm even giving a talk about how to work with 2-D data and geospatial data at PyOhio next month! I could talk at length about different granularities of shape data and their quality, but I'll just say instead it is a really exciting time to be in the geospatial data business.
posted by Alison at 3:42 PM on July 1, 2014 [6 favorites]
This is very much a calculated move. ESRI has been pitching their new Open Data portal to governmental agencies. Opening the latest ArcNews today as well and the headline is "Opening Data to the People".
posted by graxe at 4:09 PM on July 1, 2014 [1 favorite]
posted by graxe at 4:09 PM on July 1, 2014 [1 favorite]
yeah, not really a big deal...esri's hosted services let pretty much anyone publish data (depending on size & format).
things can be too big... impossible if you are in a place with limited connection speeds or old equipment...
QFT
If this results in state/local governments making their data more accessible to the public, that's huge. There are treasure troves of data out there that are just a pain in the ass to obtain and keep up to date.
I worked extensively with the business and technology behind Spatial Data Standards for Facilities, Infrastructure, and Environment (SDSFIE). The link is mainly about publishing and sharing schemas. In govt shops (dod, state, county, city, agency etc), you can stop a meeting in it's tracks by merely suggesting that any data beyond schema gets published in a public and transparent way. For too long, the (very expensive) data has been held close to the vest by the originators. We need a few more years for the generation of proprietary/closed gatekeepers to get out of the way for data sharing. ESRI wouldn't even be in the data sharing biz if Google hadn't started publishing geodata first and capturing a lot of mindshare (esri used to have ultra strict licensing around data copyright and distribution).
it is possible to take advantage of this great abundance without dealing with clunky software
The reason ESRI software is expensive is because the ArcObjects API (a set of reusable COM types and interfaces that all their products are based on) is so powerful, fine-grained, loosely-coupled, and truly object oriented. The clunky bit is whenever you have to go through specific elements of their thick-client UI, which may be inefficient Python wrappers to the ArcObjects API (e.g. crap threading).
Essentially, ArcMap is the Visual Studio of geodata manipulation. That said, it still costs way too much, by probably half. The server tier is super-awesome, but stunningly expensive, too.
it is a really exciting time to be in the geospatial data business
yep
posted by j_curiouser at 4:16 PM on July 1, 2014 [2 favorites]
things can be too big... impossible if you are in a place with limited connection speeds or old equipment...
QFT
If this results in state/local governments making their data more accessible to the public, that's huge. There are treasure troves of data out there that are just a pain in the ass to obtain and keep up to date.
I worked extensively with the business and technology behind Spatial Data Standards for Facilities, Infrastructure, and Environment (SDSFIE). The link is mainly about publishing and sharing schemas. In govt shops (dod, state, county, city, agency etc), you can stop a meeting in it's tracks by merely suggesting that any data beyond schema gets published in a public and transparent way. For too long, the (very expensive) data has been held close to the vest by the originators. We need a few more years for the generation of proprietary/closed gatekeepers to get out of the way for data sharing. ESRI wouldn't even be in the data sharing biz if Google hadn't started publishing geodata first and capturing a lot of mindshare (esri used to have ultra strict licensing around data copyright and distribution).
it is possible to take advantage of this great abundance without dealing with clunky software
The reason ESRI software is expensive is because the ArcObjects API (a set of reusable COM types and interfaces that all their products are based on) is so powerful, fine-grained, loosely-coupled, and truly object oriented. The clunky bit is whenever you have to go through specific elements of their thick-client UI, which may be inefficient Python wrappers to the ArcObjects API (e.g. crap threading).
Essentially, ArcMap is the Visual Studio of geodata manipulation. That said, it still costs way too much, by probably half. The server tier is super-awesome, but stunningly expensive, too.
it is a really exciting time to be in the geospatial data business
yep
posted by j_curiouser at 4:16 PM on July 1, 2014 [2 favorites]
Thank you for the post, I'd enjoy more metafilter discussion on GIS. I'm lucky enough to work for a non-profit using exclusively open source tools including QGIS, PostGIS, Openlayers and GeoDjango. The quality of open source and free(dom) GIS software is really good and getting better all the time. A not insignificant part of the work I do is scraping data from our local government's various public facing ESRI web products and combining it in an actually useful ways. Often the people who get the most use out of the resulting data sets is the very same local government that is paying $hit tons for ESRI's products but not getting what they actually need.
posted by ChrisHartley at 5:57 PM on July 1, 2014 [1 favorite]
posted by ChrisHartley at 5:57 PM on July 1, 2014 [1 favorite]
2014 and you still can't convert a floating-point vector to a raster
You can't up or download data from a fucking Garmin without going through some freeware, which esri breaks with every upgrade. In 2014. Fuck you ESRI.
This is really fluffy, most agencies will happily give you data in esri formats right now. I'll send you my data in a geodatabase for free! But increasingly people are just asking for .kmz and .kml files as Google earth is free and 99% of people only want to look at data, not analyze it.
posted by fshgrl at 6:00 PM on July 1, 2014 [1 favorite]
You can't up or download data from a fucking Garmin without going through some freeware, which esri breaks with every upgrade. In 2014. Fuck you ESRI.
This is really fluffy, most agencies will happily give you data in esri formats right now. I'll send you my data in a geodatabase for free! But increasingly people are just asking for .kmz and .kml files as Google earth is free and 99% of people only want to look at data, not analyze it.
posted by fshgrl at 6:00 PM on July 1, 2014 [1 favorite]
Probably important to note that Esri's been playing the closed-format game for years, introducing intentionally encrypted formats, even when open alternatives are already established. In the few cases where they've tried to play open, they've failed, because of Office XML-style standardizing-proprietary-things.
Now that we have this out of the way: this is a good effort and they're doing the right thing, and have most of the technical details right. If you're wondering what sites use this, see Data Driven Detroit. Solid APIs that serve good, simple formats. I hope more municipalities hit that button.
Full disclosure: I work for a bloodthirsty competitor of Esri and have a fun-size axe to grind.
posted by tmcw at 6:22 PM on July 1, 2014 [18 favorites]
Now that we have this out of the way: this is a good effort and they're doing the right thing, and have most of the technical details right. If you're wondering what sites use this, see Data Driven Detroit. Solid APIs that serve good, simple formats. I hope more municipalities hit that button.
Full disclosure: I work for a bloodthirsty competitor of Esri and have a fun-size axe to grind.
posted by tmcw at 6:22 PM on July 1, 2014 [18 favorites]
I work for a bloodthirsty competitor of Esri and have a fun-size axe to grind.
If thought that competitor might have been Manifold or Mapinfo and was about to yell at you, but then I browsed your Github and figured it out, and I'll just say well done, keep up the great work.
posted by Jimbob at 6:33 PM on July 1, 2014 [4 favorites]
If thought that competitor might have been Manifold or Mapinfo and was about to yell at you, but then I browsed your Github and figured it out, and I'll just say well done, keep up the great work.
posted by Jimbob at 6:33 PM on July 1, 2014 [4 favorites]
I'm a GIS professional - a cartographer/geospatial analyst, I have written about maps and cartographic issues for a few internet publications - this coincides with what appears to be a move to hamstring exporting ArcGIS to other formats. I agree with schmod and rockindata. This is a calculated ploy and should be viewed through a skeptical lens.
If this is a post made to educate people about GIS: those looking to learn more about it, look to OpenStreetMap, Mapbox, and similar projects. That's where the industry is headed. This is just a press release.
posted by troika at 6:34 PM on July 1, 2014 [6 favorites]
If this is a post made to educate people about GIS: those looking to learn more about it, look to OpenStreetMap, Mapbox, and similar projects. That's where the industry is headed. This is just a press release.
posted by troika at 6:34 PM on July 1, 2014 [6 favorites]
Hey, I use GIS at my job, too! We use community commons - for example, here's the population within .5 miles of a park
posted by rebent at 6:38 PM on July 1, 2014 [1 favorite]
posted by rebent at 6:38 PM on July 1, 2014 [1 favorite]
> … namely Python packages like Shapely and Fiona
Shapely and Fiona are such incredible packages. Sure, the former is really just an interface to GEOS, but it allows really simple access to 2D geometry tools. Outside the GIS realm, I've seen it used for structured data extraction from FOIA'd tax form scans to my own pointless diversions noodling with 1980s pen plotters. Fiona just lets you read most geo data files as familiar Python objects.
If you're digging the map things, but don't know where to start, there are Maptime groups starting in many cities.
I do take issue with the main article's ooh-this-will-be-so-good-for-openstreetmap tone. Sure, a lot of data is imported into OSM, but the project's value is in the local mappers being able to map the details that matter to them, and fix issues long before the commercial/government mapping agencies get to see them. I'm sure every municipality will pick their own precious little licence, with the unintended consequence of incompatibility if anyone should try to blend data sources. This is depicted starkly in the CIPPIC Licensing Information Project for Open Licences tool, where you can pick two or more data/licence sources and see if they're remotely compatible. They're very frequently not, which harms everyone.
posted by scruss at 5:06 AM on July 2, 2014 [1 favorite]
Shapely and Fiona are such incredible packages. Sure, the former is really just an interface to GEOS, but it allows really simple access to 2D geometry tools. Outside the GIS realm, I've seen it used for structured data extraction from FOIA'd tax form scans to my own pointless diversions noodling with 1980s pen plotters. Fiona just lets you read most geo data files as familiar Python objects.
If you're digging the map things, but don't know where to start, there are Maptime groups starting in many cities.
I do take issue with the main article's ooh-this-will-be-so-good-for-openstreetmap tone. Sure, a lot of data is imported into OSM, but the project's value is in the local mappers being able to map the details that matter to them, and fix issues long before the commercial/government mapping agencies get to see them. I'm sure every municipality will pick their own precious little licence, with the unintended consequence of incompatibility if anyone should try to blend data sources. This is depicted starkly in the CIPPIC Licensing Information Project for Open Licences tool, where you can pick two or more data/licence sources and see if they're remotely compatible. They're very frequently not, which harms everyone.
posted by scruss at 5:06 AM on July 2, 2014 [1 favorite]
Flagged as Fantastic Post.
I am so glad I am on vacation tomorrow - I have some great stuff to read - my GIS is rusty (a bit learned in my Info Sci master's program and a bit on the job years ago) and these links in the post and comments are fabulous.
posted by pointystick at 6:32 AM on July 2, 2014
I am so glad I am on vacation tomorrow - I have some great stuff to read - my GIS is rusty (a bit learned in my Info Sci master's program and a bit on the job years ago) and these links in the post and comments are fabulous.
posted by pointystick at 6:32 AM on July 2, 2014
Also, this is maybe my favorite kind of post: OP makes good post; other people with various backgrounds add more links from more POVs and the thread just gets awesomer.
posted by pointystick at 6:34 AM on July 2, 2014 [1 favorite]
posted by pointystick at 6:34 AM on July 2, 2014 [1 favorite]
So yes this is good news, in the same way that Microsoft releasing Internet Explorer 4 in 1997 was good news. Congratulations, ESRI, for finally catching up to where the rest of the mapping industry has been for the past 10 years! Now, shudder as the lumbering monopolistic behemoth makes just enough of a progressive step that it will bolster their monopoly position for a few more years. Let's just hope that they don't set the whole industry back for so many years the way IE's dominance did.
First, in the US, government data is open data. In many circumstances, particularly federal, all government-produced data is free from copyright or else freely available under some sort of required public license. But historically that GIS data has been difficult to get, because it's locked inside some Internet-ignorant ESRI systems with proprietary formats and insane, horrible server implementations. Ever gone to a website to "prepare a download" of 100 MB of data that will then be available 24 hours later, as if some tape robot had to find it? I don't know that an ESRI system is always to blame for that nonsense, but it sure fits their modus operandi.
ESRI has improved the past few years, once they saw some competition from things like Google Maps and the open source mapping world. For instance they caught up last year to 2005-era Google Maps, letting you publish dots on a map. The web deliverable is comically bad; 750k of unminified files, about 50 separate HTTP transactions, with absurd trade secret declarations in the comments. By contrast I could knock that map out in about an hour with open source Leaflet at 1/10 the size, 3x the speed, and greater usability. For free. ESRI's stuff has value in that it's an integrated suite with easy to use tools. But it's expensive, proprietary, and mediocre.
So anyway, it's great that ESRI is crawling forward to allow government agencies to finally publish the data under the licenses they should have been using for years. But we'd all be better off if they just stayed slumbering, eventually dying in their sleep. Meantime their competition is improving and doing a lot of really innovative, truly open work. MapBox has a particularly strong set of products.
If this stuff interests you, come to the FOSS4G in Portland in September 2014, the open source mapping conference. I went a couple of years ago and got a fantastic education in the leading edge of mapping tech, looking forward to going back and learning what's new.
posted by Nelson at 7:32 AM on July 2, 2014 [8 favorites]
First, in the US, government data is open data. In many circumstances, particularly federal, all government-produced data is free from copyright or else freely available under some sort of required public license. But historically that GIS data has been difficult to get, because it's locked inside some Internet-ignorant ESRI systems with proprietary formats and insane, horrible server implementations. Ever gone to a website to "prepare a download" of 100 MB of data that will then be available 24 hours later, as if some tape robot had to find it? I don't know that an ESRI system is always to blame for that nonsense, but it sure fits their modus operandi.
ESRI has improved the past few years, once they saw some competition from things like Google Maps and the open source mapping world. For instance they caught up last year to 2005-era Google Maps, letting you publish dots on a map. The web deliverable is comically bad; 750k of unminified files, about 50 separate HTTP transactions, with absurd trade secret declarations in the comments. By contrast I could knock that map out in about an hour with open source Leaflet at 1/10 the size, 3x the speed, and greater usability. For free. ESRI's stuff has value in that it's an integrated suite with easy to use tools. But it's expensive, proprietary, and mediocre.
So anyway, it's great that ESRI is crawling forward to allow government agencies to finally publish the data under the licenses they should have been using for years. But we'd all be better off if they just stayed slumbering, eventually dying in their sleep. Meantime their competition is improving and doing a lot of really innovative, truly open work. MapBox has a particularly strong set of products.
If this stuff interests you, come to the FOSS4G in Portland in September 2014, the open source mapping conference. I went a couple of years ago and got a fantastic education in the leading edge of mapping tech, looking forward to going back and learning what's new.
posted by Nelson at 7:32 AM on July 2, 2014 [8 favorites]
Man, these are some great comments.
I've only ever used Esri products so my view of the GIS world is pretty limited. I had no idea there was so much exciting stuff happening on the outside... hope I can find some time to fall down this rabbit hole soon.
posted by whitecedar at 8:16 AM on July 2, 2014
I've only ever used Esri products so my view of the GIS world is pretty limited. I had no idea there was so much exciting stuff happening on the outside... hope I can find some time to fall down this rabbit hole soon.
posted by whitecedar at 8:16 AM on July 2, 2014
Wow, there is so much to pick through here, and I haven't had the chance yet, but I will! I work in Open Data, so this is very professionally relevant to me. I made my first step into mapping data earlier this year, making a .kml file of crowdsourced data for a side project; I wanted to use Open Street Map but just couldn't get it to work. I'm happy for the chance to pick up some knowledge from everyone above this comment with more experience than me. :)
rockindata: "The federal government is moving in a number of directions at once, but things really do seem to be focusing on data.gov, which is moving to an open-source, standards-compliant cataloging tool called CKAN."
Yay, CKAN! It's a very cool tool and has a good dev community around it.
Alison: " I use free software tools to work with these shapes, namely Python packages like Shapely and Fiona."
*takes note of package names*
posted by daisyk at 1:50 PM on July 2, 2014
rockindata: "The federal government is moving in a number of directions at once, but things really do seem to be focusing on data.gov, which is moving to an open-source, standards-compliant cataloging tool called CKAN."
Yay, CKAN! It's a very cool tool and has a good dev community around it.
Alison: " I use free software tools to work with these shapes, namely Python packages like Shapely and Fiona."
*takes note of package names*
posted by daisyk at 1:50 PM on July 2, 2014
(Full disclosure, I have made a couple of pull requests to the CKAN codebase and use it for my job. It's part of my job that I really enjoy, though!)
posted by daisyk at 1:55 PM on July 2, 2014
posted by daisyk at 1:55 PM on July 2, 2014
tmcw is being too modest, as he's responsible for the small but amazing mapschool (previously) and the data sharing via github site geojson.io.
If you just want to draw some stuff on a map, share it, save it, reuse it ... then umap is your friend.
posted by scruss at 5:49 PM on July 2, 2014 [3 favorites]
If you just want to draw some stuff on a map, share it, save it, reuse it ... then umap is your friend.
posted by scruss at 5:49 PM on July 2, 2014 [3 favorites]
The web deliverable is comically bad...ESRI's stuff has value in that it's an integrated suite...But it's expensive, proprietary, and mediocre.
So I'm not an ESRI apologist, but you're missing some of the pretty awesome forest for the (esri client app) trees. Yeah, all the clients except the flagship ArcMap totally blow. But you are not their target market - those clients are for non-programmers who just need to get something to 'run'.
For instance they caught up last year to 2005-era Google Maps, letting you publish dots on a map.
You are off-base here. ArcGIS Online has allowed geodata publishing for years. If you want to publish on a large scale or with very specific cartography and capabilities, the server tier is pretty sweet. Also, even in that horrible JavaScript API, you can create all sorts of features in various geometries - and have been able to for years.
Don't conflate different elements of the suite - it's like saying C# sucks because MS Access does.
posted by j_curiouser at 7:26 PM on July 2, 2014
So I'm not an ESRI apologist, but you're missing some of the pretty awesome forest for the (esri client app) trees. Yeah, all the clients except the flagship ArcMap totally blow. But you are not their target market - those clients are for non-programmers who just need to get something to 'run'.
For instance they caught up last year to 2005-era Google Maps, letting you publish dots on a map.
You are off-base here. ArcGIS Online has allowed geodata publishing for years. If you want to publish on a large scale or with very specific cartography and capabilities, the server tier is pretty sweet. Also, even in that horrible JavaScript API, you can create all sorts of features in various geometries - and have been able to for years.
Don't conflate different elements of the suite - it's like saying C# sucks because MS Access does.
posted by j_curiouser at 7:26 PM on July 2, 2014
(also, MapBox rules)
posted by j_curiouser at 7:29 PM on July 2, 2014
posted by j_curiouser at 7:29 PM on July 2, 2014
If you just want to draw some stuff on a map, share it, save it, reuse it ... then umap is your friend.
Hmm umap is nice - shame it doesn't support Shapefile import, but that's okay, you can convert a Shapefile to GeoJSON in 4 lines of R code.
posted by Jimbob at 9:01 PM on July 2, 2014
Hmm umap is nice - shame it doesn't support Shapefile import, but that's okay, you can convert a Shapefile to GeoJSON in 4 lines of R code.
posted by Jimbob at 9:01 PM on July 2, 2014
« Older 20,000 voices, singing as one | an overconfident autodidact’s imitation of a Lewis... Newer »
This thread has been archived and is closed to new comments
posted by mlle valentine at 1:55 PM on July 1, 2014 [2 favorites]