Tag: GIS

How to convert GTFS to GIS shapefiles and KML

October 12, 2010 / / 41 Comments

This tutorial will teach how you to convert any transit agency’s General Transit Feed Specification (GTFS) data into ESRI ArcGIS-compatible shapefiles (.shp), KML, or XML. This is simple to do because GTFS data is essentially a collection of CSV (comma separated values) text files (really, really large text files).

Note: I don’t know how to do the reverse, converting shapefiles or other geodata into GTFS data. I’m not sure if this is possible and I’m still investigating it. If you have tips, let me know.

Converting GTFS to GIS shapefiles

Instructions require the use of ArcGIS (Windows only) and a free plugin called ET GeoWizards GIS for any version of ArcGIS. I do not have instructions for Mac users at this time.

I wrote these instructions while converting the Chicago Transit Authority’s GTFS files into shapefiles based on a reader’s request. “Field names” are quoted and layer names are italicized.

  1. Download the GTFS data you want. Find data from agencies around the world (although not many from Europe) on GTFS Data Exchange.
  2. Import into ArcGIS the shapes.txt file using Tools>Add XY Data. Specify Y=lat and X=lon
  3. Using ET GeoWizards GIS tools, in the Convert tab, convert the points shapefile to polyline.
  4. Select the shapes layer in the wizard, then create a destination file. Click Next.
  5. Select the “shape_id” field
  6. Click the checkbox next to Order and select the field “shape_pt_sequence” and click Finish.
  7. Depending on the number of records (the CTA has 466,000 shapes), it may take a while.
  8. The new shapefile will be added to your Table of Contents and appear in your map.
  9. Import the trips.txt and routes.txt files. Inspect them for any NULL values in the “route_id” field. You will be using this field to join the routes and trips table. It may be a case that ArcGIS imported them incorrectly; the text files will show the correct data. If NULL values appear, follow steps 10 and 11 and continue. If not, follow steps 10 and 12 and continue. This happens because ArcGIS inspected some of the data and determined they were integers and ignored text. However, this is not the case.
  10. Export the text files as DBF files so that ArcGIS operates on them better. Then remove the text files from the Table of Contents.
  11. (Only if NULL values appear) Go into editing mode and fix the NULL values you noticed in step 9. You may have to make a new column with a more forgiving data type (string) and then copy the “route_id” column into the new column. Then continue to step 12.
  12. Join routes and trips based on the field “route_id” – export as trips_routes.dbf
  13. Add a new column to shapes.shp called “shape_id2”, with data type double 18, 11. This is so we can perform step 14. Use the field calculator to copy the values from “shape_id” (also known as ET_ID) to “shape_id2”
  14. Join routes_trips with shapes into routes_poly based on the field “shape_id” (and “shape_id2”)
  15. Dissolve routes_poly on “route_id.” Make sure all selections are cleared. Use statistics/summary fields: “route_long,” “route_url.” Save as routes_diss.shp
  16. Inspect the new shapefile to ensure it was created correctly. You may notice that some bus routes don’t have names. Since these routes are well documented on the CTA website, I’m not going to fill in their names.

Click on the screenshot to see various steps in the tutorials.

Converting GTFS to KML

After you have it in shapefile form, converting to KML is easy – follow these instructions for using QGIS. Or if you want to skip the shapefile-creation process (quite involved!), you can use KMLWriter, a Python script. Also, I think the latest version of ArcGIS has built-in KML exporting.

Converting GTFS to XML

If you want to convert the GTFS data (which are essentially comma-separated value – CSV – files) to XML, that’s easier and you can avoid using GIS programs.

  • First try Mr. Data Converter (very user friendly).
  • If that doesn’t work, try this website form on Creativyst. I tested it by converting the CTA’s smallest GTFS table, frequencies.txt, and it worked properly. However, it has a data size limit. (User friendly.)
  • Next try csv2xml, a command line tool. (Not user friendly.)
  • You can also use Microsoft Excel, but read these tips and caveats first. (I haven’t found a Microsoft application I like or think is user friendly.)

How to geocode a single address in QGIS

October 3, 2010 / / 0 Comments

Since the last time I wrote about how to use BatchGeocode.com to perform pseudo-geocoding tasks in QGIS, there have been considerable improvements in the multi-platform, free, and open source GIS software. Now, geocoding (turning addresses into coordinates) is more automatic, albeit difficult to setup. (Okay, this has been around June 2009 and I just found out about it in October 2010.)

Once you install all the components, you’ll never have to do this again.

This method can only geocode one address at a time, but it will geocode all of the addresses into a single shapefile.

  1. Download QGIS.
  2. Download and install Python SetupTools. This includes the easy_install function that will download a necessary Python script, simplejson. On Mac you will have to use the Terminal (Applications>Utilities). Email me if you run into problems.
  3. Install simplejson. In the command line (Terminal for Mac; in Windows press Start>Run>”cmd”>Enter), type “easy_install simplejson”.
  4. Download the GeoCode plugin by Alessandro Pasotti via QGIS>Plugins>Fetch Python Plugins. You may have to load additional repositories to see it.
  5. Install geopy. In the command line (like step 3), type “easy_install geopy”.
  6. Specify your project’s projection in File>Project Properties.
  7. Get a Google Maps API key and tell the GeoCode plugin about it (QGIS>Plugins>GeoCode>Settings). You will need a Google account. If you don’t have your own domain name, you can just enter “google.com” when it asks for your domain.
  8. Geocode your first address by clicking on Plugins>GeoCode>Geocode. Type the full address (e.g. 121 N LaSalle Street, Chicago, IL for City Hall).
  9. The geocoded address will then appear in your Layers list as its own shapefile. All addresses geocoded (or reverse geocoded) in this project will appear in the same layer (therefore same attribute table).

Once you install all the components, you’ll never have to do this again. Geocoding will be available each and every time you use QGIS in the future on that workstation.

Tips

  • When you’re done geocoding,  save your results as a shapefile (right click the layer and click “Save as shapefile”). Twice I’ve lost my results after saving the project and quitting QGIS. When I reopened the project, the results layer was still listed, but contained no data.
  • Add a “name” column to the GeoCoding Plugin Results layer’s attribute table (toggle editing first). You can then type in the name of the building or destination at the address you geocoded. Edit the layer’s properties to have that name appear as a label for the point.

A map I made with QGIS showing three geocoded points of interest in Chicago. Data from City of Chicago’s GIS team.

 

Trying out new GIS software

October 2, 2010 / / 0 Comments

I want to draw 50 and 120 feet buffers around the points of store entrances to show where bike parking should and shouldn’t be installed. I want to follow this example:

walgreens with bike parking buffers

Aerial photo of a Tucson, Arizona, Walgreens showing the location of existing bike parking and two buffers (50 and 120 feet) where proposed city rules would allow bike parking. I advocate for ratifying the 50 feet rule, which I’ve discussed on this blog and elsewhere many times.

I want to do this easily and accurately, so I will use GIS software to create a “buffer.” I use QGIS occasionally, but I want to try out other Mac-friendly applications. I’m getting my orthoimagery (geometrically corrected aerial photography) from the United States Geological Survey (USGS) using a web protocol called Web Map Server. I’m trying:

  • Cartographica, $495, with free trial license.
  • uDig, completely free software. UPDATE: I have had NO success getting any data to load from a WMS connection into uDig. I would like to understand why. Cartographica can obtain some of the WMS-stored data I want, although it messes up often.

I’m having success with neither – both are having issues downloading or maintaining a connection to the USGS orthoimagery. In one case, Cartographica trims the Bing Maps imagery to match the extent of my other objects (the buffer). In another case, it won’t even download the USGS imagery (and gives no indication that anything is happening). uDig hasn’t been able to download anything so far – I hope it’s asking for the current extent, instead of all data because it’s taking a looong time to do anything (so long that I just quit in the  middle of it).

This screenshot shows how to add new WMS connections to Cartographica.

UPDATE: I did it! I successfully used Cartographica (and the integrated Bing Maps) to create this drawing that shows the current (abysmal) bike parking at a Chicago Home Depot outside the 50 feet line.

Update on GIS information for Haiti

January 20, 2010 / / 0 Comments

We all woke up this morning to see news that another earthquake has happened in Haiti, near the center of the first one eight days ago.

“The United Nations Development Programme (UNDP) employed nearly 400 Haitians in cash-for-work activities to jump start the local economy and facilitate the delivery of urgently needed humanitarian assistance.”

This post is an update to my previous article about how GIS is used for disaster relief efforts. I recently came across a webpage on Harvard’s China Earthquake Geospatial Research Portal that lists copious, up-to-date, GIS-compatible data from organizations around the world. The portal began in response to the Sichuan, China, earthquake in May 2008.

Visit the Haiti GIS Data Portal now.

For new GIS students, this would be a great starting point for a class final project. The Portal is hosting the datasets as a public service and invites anyone with relevant data to submit it to the site operators for wider dissemination. Data comes from the United Nations, several universities, OpenStreetMap contributors, and the German Center for Air and Space Travel, among others.

“Petty Officer 3rd Class Cameron Croteau, a Damage Controlman aboard the Coast Guard Cutter Oak, carries an injured Haitian girl to an awaiting Coast Guard HH-60 Jayhawk helicopter Tuesday, Jan. 19, 2010. Coast Guard and Navy helicopters airlifted injured Haitians to a private hospital in Milot, Haiti. U.S. Coast Guard photo by Petty Officer 3rd Class Brandyn Hill.”

As I mentioned in the previous post, there are many photos on Flickr when you search for “haiti earthquake.” When I wrote the post on January 14, 2010, there were only about 300 photos, and now there are over 6,900. Only 1,200 have a Creative Commons license, though (both of the photos above have a Creative Commons license). It seems that the United States Military, the United Nations, and major relief organizations are providing the majority of photos. And they’re uploading them fast. The number of photos on Flickr jumped by 50 from when I started this paragraph.

How GIS helps earthquake relief efforts for Haiti

January 14, 2010 / / 0 Comments

While Geographic Information systems software can definitely produce pretty maps, its power lies in analyzing data and plotting or comparing sensory or observed data to spatial data (like roads or terrain). The earthquake in Haiti rocked the capital city, Port-au-Prince with a shock of magnitude 7.0 on Tuesday, January 12, 2010.

A photo from a United States military flyover shows damage in the Port of Port-au-Prince, Haiti. Photo taken by Petty Officer 2nd Class Sondra-Kay Kneen and uploaded by Chuck Simmins.

There are several applications for GIS to help with earthquake response, and two blog posts that appeared this morning shed light on how.

The first article came from ESRI, the California-based makers of ArcGIS, the most used GIS application. The article linked to a user-built map on their ArcGIS Online service showing on Bing maps where the earthquake and its aftershocks struck (the map sits behind a registration wall). ESRI even has a disaster response team that helps organizations get their response projects off the ground quickly.

Infrastructurist posted the second article, showing some before and after satellite imagery of Haiti, provided by Google and GeoEye.

So what can GIS do? From ESRI’s list, “GIS for Disaster Response“:

  • Rapid identification of potential shelter/housing locations (schools, libraries, churches, public buildings) appropriate for supporting affected populations.
  • Determine how many tents will be needed based on the location of populations affected by the disaster.
  • Analyze areas where large numbers of refugees can establish camps out of harm’s way that are accessible for supply delivery and have access to water and other resources necessary to support large numbers of people.
  • Many more examples.

Want more information? Here’s where to get it: