Tag: tutorial

Download GIS data from ArcGIS servers using a simple command

October 21, 2017 / / 0 Comments

A lot of geospatial data (GIS) in the world is stored on ArcGIS MapServers, a part of the Esri “stack” of products that municipalities and other governments use to manage and publish GIS data.

A lot of people want that data, and not every government has enabled the ArcGIS data hub features that make it easier to get. For an example data hub, check out Cook County, Illinois.

If you have ArcGIS software on your Windows computer, then it can be pretty easy to plug in the map server URL and manipulate and extract the data.

For the rest of us who don’t have a very expensive license for that software, you can use a “command line” tool (written in Python) on any computer to download any layer of GIS data hosted on the ArcGIS MapServer and automatically convert it to GeoJSON.

Tutorial

You’ll need to install the Python package pyesridump, from the OpenAddresses GitHub repository, created by Ian Dees and other contributors.

Installing pyesridump is easy if you have pip installed, using the command pip install esridump.

The next thing you’ll need is the URL to a layer in a MapServer, and these are not easy to find.

Finding data to download

I can almost guarantee the county where you live has one. Before you continue, check to see if your county (or other jurisdiction) has the “open data portal” add-on to their ArcGIS stack.

A handful of suggestions:

  • Here are links to the open data portals enabled by Esri for Lake County, Illinois, and Broomfield County, Colorado.
  • A reader suggested looking for MapServers for jurisdictions around the world by looking through Esri’s portal of open data called ArcGIS Hub.
  • Use a search engine to search for the URL pattern that’s common to ArcGIS servers: paste “inurl:arcgis/rest/services” into a search engine. (You can also limit this search to a particular domain name, say, “chicago.gov”, by adding “site:chicago.gov” to the search query, making the full search query “inurl:arcgis/rest/services site:chicago.gov”.

Once you locate a dataset you want, you can find the MapServer URL under About>Data Source on the right side of the page.

I normally find them by looking at the HTML source code of a MapServer I already know about.

For this example I’ll use one of the GIS layers in the Cook County, Illinois, election service MapServer – here’s the layer for the Cook County commissioners districts.

Fetch the data

Once you have the URL the command is simple:

esri2geojson http://cookviewer1.cookcountyil.gov/ArcGIS/rest/services/cookElectnSrvc/MapServer/11 cookcounty_commissioners.geojson

  • The first term, esri2geojson tells your computer which program to load.
  • The second term is the URL of the MapServer URL.
  • The third term is the filename and location where you want to store the file. I prefer running the command “inside” the folder where I want the file to be stored. You can also specify a full path of the file. On a Mac this would look like ~/Users/username/Documents/GIS/projectname/cookcounty_commissioners.geojson

After you enter the command into your computer’s terminal, press enter. esri2geojson will report back once, after it finds and understands the MapServer URL you gave it. When it’s done, the command will “close” and your computer’s terminal will wait for the next command.

Do you have questions, or need some help? Leave a comment below.

Converting shapefiles to GeoJSON, and other format conversions

November 12, 2012 / / 0 Comments

To develop the Chicago Bike Map app, I had a problem I thought would be simple to solve: load train lines into a Leaflet-powered map. I had the train lines stored as a polyline shapefile but Leaflet can only read the GeoJSON format or a string of geographic coordinates representing lines.

I eventually found a solution (I can’t remember how) and I need to share it with you. The converter can do more than ESRI shapefiles to GeoJSON. It can reproject the data in the conversion. It can convert from several formats to several other formats.

The site is called MyGeodata Converter. You upload a ZIP file of geographic files – .shp and its companion files (.prj, .dbf, .shx), .kml, and .gpx. Let’s take the Chicago Transit Authority train lines shapefile straight from the City of Chicago’s open data portal. It downloads as a zipped collection of a shapefile and its buddies and we can take this file straight to the Converter and upload it. The Converter will unzip it and read the data; it will even identify the projection system (for Chicago-based geographic data, its common to use NAD83 Illinois StatePlane East FIPS 1201 Feet (SRID 102671, the same as SRID 3435).

The Converter will convert to one of the following formats, with same or new projection; accepts SQL statements to extract a subset of data:

  • ESRI shapefile
  • GML
  • KML, KMZ
  • GeoJSON
  • Microstation DGN
  • MapInfo File
  • GPX
  • CSV

New map tutorials

February 14, 2012 / / 0 Comments

A screenshot of using BatchGeocode to take a spreadsheet of addresses and turn it into a nice map. 

Over at Grid Chicago, my other blog that sucks all the time from this blog, I’ve recently written tutorials on how to create online maps, first with Google My Maps (which they renamed to My places) and secondly with BatchGeocode (which renamed itself to BatchGeo because it does more than geocoding now).

Google My Maps is primitive as far as map making goes, but it has the lowest learning curve and it’s easy: you just click on the map where you want something to go and fill in the info window. Read that tutorial.

BatchGeocode is slightly more advanced, but takes your tabular data (most likely from a spreadsheet) and throws it on a map you can embed on your website. They do have pay features. Read the tutorial for BatchGeocode.

I’ve written about BatchGeocode for QGIS, as it was once the only way to do geocoding in QGIS. But now BatchGeocode doesn’t give you a results table that has the latitude and longitude (apparently this is against Google Maps’s terms of service). But I updated the article to talk about using other methods for geocoding in QGIS.

I will be writing two more tutorials, one about GeoCommons and one about Google Fusion Tables.

How to upload shapefiles to Google Fusion Tables

September 25, 2011 / / 0 Comments

It is now possible to upload a shapefile (and its companion files SHX, PRJ, and DBF) to Google Fusion Tables (GFT).

Before we go any further, keep in mind that the application that does this will only process 100,000 rows. Additionally, GFT only gives each user 200 MB of storage (and they don’t tell you your current status, that I can see).

  1. Login to your Google account (at Gmail, or at GFT).
  2. Prepare your data. Ensure it has fewer than 100,000 rows.
  3. ZIP up your dataX.shp, dataX.shx, dataX.prj, and dataX.dbf. Use WinZip for Windows, or for Mac, right-click the selection of files and select “Compress 4 items”.
  4. Visit the Shape to Fusion website. You will have to authorize the web application to “grant access” to your GFT tables. It needs this access so that after the web application processes your data, it can insert it into GFT.
  5. If you want a Centroid Geometry column or a Simplified Geometry column added, click “Advanced Options” and check their checkboxes – see notes below for an explanation.
  6. Choose the file to upload and click Upload.
  7. Leave the window open until it says it has processed all of the rows. It will report “Processed Y rows and inserted Y rows”. You will be given a link to the GFT the web application created.

Sample Data

If you’re looking to give this a try and see results quickly, try some sample data from the City of Chicago data portal:

Notes

I had trouble many times while using Shape to Fusion in that after I chose the file to upload and clicked Upload, I had to grant access to the web application again and start over (choose the file and click Upload a second time).

Centroid Geometry – This creates a column with the geographic coordinates of the centroid in a polygon. It lists it in the original projection system. So if your projection is in feet, the value will be in feet. This is a function that can easily be performed in free and open source QGIS, where you can also reproject files to get latitude and longitude values (in WGS84 project, EPSG 4326). The centroid value is surrounded in the field by KML syntax “<Point><coordinates>X,Y</coordinates></Point>”.

Simplified Geometry – A geometry column is automatically created by the web application (or GFT, I’m not sure). This function will create a simpler version of that geometry, with fewer lines and vertices. It also creates columns to list the vertices count for the simple and regular geometry columns.

Just how many taxi vs. bicycle crashes are there? A Google Refine story

April 26, 2011 / / 0 Comments

On this Chicagoist story about how IDOT will now collect data on doorings (instead of ignoring that crash type as they preferred), I opened the story photo entitled “Cabbie takes down another” by Moe Martinez. His photo caption reads, “you see it alot … thankfully this guy seemed to be relatively ok … coherent and what not.”

I wanted to know just how often “we” see taxi drivers crashing with people riding bicycles. You can’t filter by vehicle type in either mine or Derek’s bike crash maps, but you can via the Fusion Table.

I decided to get the answer via Google Refine and make a screencast to show you just how quick and powerful a tool it is.

It’s dead simple:

  1. Load a CSV of the data into Google Refine.
  2. Click on the VEH1_SPECL column’s down arrow, then Facet>Text Facet.
  3. In the facet box, sorted alphabetically, find “TAXI/FORE HIRE.”
  4. The number of rows that apply is listed: 353.
  5. Divide 353 by the total number of rows, 4931, multiply by 100, and you get your percentage.

Taxi drivers are involved in just 7.2% of bicycle crashes in Chicago in 2007-2009.

The majority of crashes, at 66%, involve people driving “PERSONAL” vehicles. And 80% of those crashes are with a passenger vehicle that’s not a van, minivan, SUV, truck, or bus (so probably a sedan or coupe). Let’s look at more data.

How many taxis are there and how many personal vehicles are there? Are taxicabs involved in a disproportionately higher number of crashes?

About 781,023 people drive to work, either alone or with someone else, in Chicago (data from 2005-2009 5-year American Community Survey). 1,063,047 households have 1,218,594+ vehicles available in Chicago. Let’s assume the 7,000 taxicabs in Chicago are not counted as a “vehicle available.”* That’s 1,225,594 “personal” vehicles. If all were on the road at the same time, only 0.57% of them would be taxicabs. But they’re not on the road at the same time. So let’s take that number of people who drive to work and add 7,000 vehicles to it. So of those 788,023 “vehicles” now on the road, just 0.88% of them are taxicabs.

So it does seem that taxicabs are involved in a disproportionate number of crashes when compared to their presence on the streets. However, taxicabs are most likely driven more more miles and for more time than personal vehicles thus making their exposure to people bicycling greater than drivers of other vehicles. (A majority of “personal” trips are very short.)

New data coming soon

I can’t wait to get the 2010 crash data. Here’s why: In 2007, students in a taxi driver training course at Harold Washington College received some education about sharing the road with bicyclists:

A pilot “Share The Road” education module was launched at the taxi training school at Harold Washington College. It includes a 25-30 minute lecture, with discussion. After the pilot, the class will be required for all people training to drive taxis in Chicago. In the future, bicycle questions will be included on the exams required to become taxi drivers. June 2007 MBAC meeting minutes (PDF).

The number of crashes between taxi drivers and people riding bikes jumped from 2007 to 2008, but declined heavily between 2008 and 2009. More data will show us a clearer trend that may lend insight into the impact of the “Share The Road” education module.

*Notes

The question (PDF) on the American Community Survey asks, “How many automobiles, vans, and trucks of one-ton capacity or less are kept at home for use by members of this household?” This may or may not include taxicabs stored at home.

I don’t know how many taxicabs there are in Chicago, but the Chicago Sun-Times reported there are approximately 7,000.