Tag: GTFS

The U.S. DOT should collaborate with existing “National Transit Maps” makers

March 31, 2016 / / 1 Comment

The U.S. DOT demonstrated one idea for how a National Transit Map might look and work at a conference in February.

The Washington Post reported this month that the United States Department of Transportation is going to develop a “National Transit Map” because, frankly, one doesn’t exist. The U.S. DOT said such a map could reveal “transit deserts” (the screen capture above shows one example from Salt Lake City, discussed below).

Secretary Anthony Foxx wrote in an open letter to say that the department and the nation’s transit agencies “have yet to recognize the full potential” of a data standard called the General Transit Feed Specification that Google promoted in order to integrate transit routing on its maps. Foxx described two problems that arose out of not using “GTFS”.

  1. Transit vehicles have significantly greater capacity than passenger cars, but are often considered just vehicles because we are unable to show where and when the transit vehicles are scheduled to operate. The realistic treatment of transit for planning, performance measures, and resiliency requires real data on transit system operations.
  2. One of the most important social values of transit is that it makes transportation available to people who do not have access to private automobiles, and provides transportation options for those who do. Yet, we cannot describe this value at a national level and in many regions because we do not have a national map of fixed transit routes.

“The solution is straightforward”, Foxx continued, “[is] a national repository of voluntarily provided, public domain GTFS feed data that is compiled into a common format with data from fixed route systems.”

The letter went on to explain exactly how the DOT would compile the GTFS files, and said the first “collection day” will be March 31, this week. As of this writing, the website to which transit agencies must submit their GTFS files is unavailable.

What Foxx is asking for has already been done to some degree. Two national transit maps and one data warehouse already exist and the DOT should engage those producers, and others who would use the map, to determine the best way to build a useful but inexpensive map and database. Each of the two existing maps and databases was created by volunteers and are already-funded projects so it would make sense to maximize the use of existing projects and data.

“Transitland” is a project to host transit maps and timetables for transit systems around the world. It was created by Mapzen, a company funded by Samsung to build open source mapping and geodata tools. Transitland is also built upon GTFS data from agencies all over the world. Its data APIs and public map can help answer the question: How many transit operators serve Bay Area residents, and what areas does each service?

For the United States, Transitland hosts and queries data from transit agencies in 31 states and the District of Columbia. In Washington, D.C., Transitland is aware of four transit agencies. It’s a great tool in that respect: Not all of the four transit agencies are headquartered in D.C. or primarily serve that city. The app is capable of understanding spatial overlaps between municipal and regional geographies and transit agencies.

Transitland has a “GUI” to show you how much transit data it has around the world.

“Transit Explorer” is an interactive map of all rail transit and bus rapid transit lines in the United States, Mexico, and Canada. Yonah Freemark, author of The Transport Politic, created the map using data culled from OpenStreetMap, the National Transit Atlas Database (administered by the DOT and which shows fixed-guideway transit), and his own research. I wrote the custom JavaScript code for the Leaflet-powered map.

No other agency or project has collected this much data about fixed-guideway transit lines in any of the three countries, since the map includes detailed information about line lengths, ridership, and other characteristics that are not included in GTFS data. Transit Explorer, though, does not include local bus service or service frequencies, which the DOT’s map may if it incorporates the full breadth of GTFS data.

Transit Explorer also goes a step further by providing data about under construction and proposed fixed-guideway transit lines, which is information that is very relevant to understanding future neighborhood accessibility to transit, but which is not available through GTFS sources.

Finally, “GTFS Data Exchange” is a website that has been storing snapshots of GTFS feeds from agencies around the world for almost a decade, or about as long as GTFS has been used in Google Maps. The snapshots allow for service comparisons of a single agency across time. For example, there are over 100 versions of the GTFS data for the Chicago Transit Authority, stretching back to November 2009; new versions are added – by “cta-archiver” – twice a month.

Josh Cohen, writing in Next City, highlighted the significance of Google’s invention of GTFS, saying, “Prior to the adoption of GTFS, creating such a map would’ve been unwieldy and likely produced an out-of-date product by the time it was completed.” The DOT’s own National Transit Atlas Database includes only fixed-guideway (a.k.a. trains) routes, and hasn’t been updated since 2004.

Not all GTFS feeds are created equal, though. Some transit agencies don’t include all of the data, some of which is optional for Google Map’s purpose, that would make the National Transit Map useful for the spatial analysis the DOT intends. Many agencies don’t include the “route shapes”, or the geographic lines between train stations and bus stops. Researchers are able to see where the vehicles stop, but not which streets or routes they take. Foxx’s letter doesn’t acknowledge this. It does, however, mention that transit agencies can use some federal funds to create the GTFS data.

David Levinson, professor at the University of Minnesota, believes the map will bias coverage (geographic reach of transit service) over frequency (how many buses are run each day that someone could ride).

The U.S. DOT’s chief data officer, Dan Morgan, whom I met at Transportation Camp 2015 in Washington, D.C., presented at the FedGIS Conference this year one idea to demonstrate coverage and frequency in Salt Lake City, using the GTFS data from the Utah Transit Authority.

Levinson also tweeted that it will be difficult for a national map to show service because of the struggles individual transit providers have symbolizing their own service patterns.

Foxx’s letter doesn’t describe how planners will be able to download the data in the collection, but whichever app they build or modify will cost money. Before going much further, and before spending any significant funds, Foxx should consult potential users and researchers to avoid duplicating existing projects that may ultimately be superior resources.

Foxx can also take advantage of “18F” a new agency within the General Services Administration to overcome government’s reputation for creating costly and difficult to use apps. The GSA procures all kinds of things the federal government needs, and 18F may be able to help the DOT create the National Transit Map (and database) in a modern, tech and user-friendly way – or write a good RFP for someone else to make it.

Look for the National Transit Map this summer.

Converting a transit agency’s GTFS to shapefile and GeoJSON with QGIS

February 2, 2016 / / 6 Comments

Many years ago I wrote a tutorial on how to use an ArcGIS plugin to convert a transit agency’s GTFS package – a group of files that describe when and where their buses and trains stop – into files that could easily be manipulated by popular GIS desktop software.

That was so long ago, before I became an expert in using QGIS, a free and open source alternative to ArcGIS.

This tutorial will show you how to convert GTFS to a shapefile and to GeoJSON so you can edit and visualize the transit data in QGIS.

Prerequisites

First you’ll need to have QGIS installed on your computer (it works with Linux, Mac, and Windows). Second you’ll need a GTFS package for the transit agency of your choice (here’s the one for Pace Suburban Bus*, which operates all suburban transit buses in Chicagoland). You can find another transit agency around the world on the GTFS Data Exchange website.

Section 1: Let’s start

  1. Open QGIS.
  2. Load your GTFS data into the QGIS table of contents (also called the Layers Panel). Click Layer>Add Layer>Add Delimited Text Layer. You will be adding one or two files depending on which ones are provided.

    QGIS add delimited text layer

    Add delimited text layer.

  3. Now, here it can get tricky. Not all transit agencies provide a “shapes.txt” file. The shapes.txt file draws out the routes of buses and trains. If it’s not provided, that’s fine, but if you turn them into routes based on the stops.txt data, then you will have funny looking and impossible routes.

    QGIs browse for the stops.txt file

    Browse for the stops.txt file

  4. Click on “Browse…” and find the “stops.txt”. QGIS will read the file very quickly and determine which fields hold the latitude and longitude coordinates. If its determination is wrong, you can choose a different “X field” (longitude) and “Y field” (latitude).
  5. Click “OK”. A new dialog box will appear asking you to choose a coordinate reference system (EPSG). Choose or filter for “WGS 84, EPSG:4326”. Then click “OK”.
  6. The Pace bus stops in the Chicagoland region are now drawn in QGIS!

    Pace bus stops are shown

    Pace bus stops are shown

  7. If the GTFS package you downloaded includes a “shapes.txt” file (that represents the physical routes and paths that the buses or trains take), import that file also by repeating steps 4 and 5.

Section 2: Converting the stops

It’s really easy now to convert the bus or train stops into a shapefile or GeoJSON representing all of those points.

  1. Right-click the layer “stops” in the table of contents (Layers Panel) and click “Save As…”.
  2. In the “Save vector layer as…” dialog box, choose the format you want, either “ESRI Shapefile” or “GeoJSON”. **
  3. Then click “Browse” to tell QGIS where in your computer’s file browser you want to save the file. Leave the “CRS” as-is (EPSG:4326).

    Convert the bus stops to a shapefile or GeoJSON.

    Convert the Pace bus stops to a shapefile or GeoJSON.

  4. Then click “OK” and QGIS will quickly report that the file has been converted and saved where you specified in step 3.

Section 3: Converting the bus or train routes

The “shapes.txt” file is a collection of points that when grouped by their route number, show the physical routes and paths that buses and trains take. You’ll need a plugin to make the lines from this data.

  1. Install the plugin “Points to Paths”. Click on Plugins>Manage and Install Plugins… Then click “All” and search for “points”. Click the “Points to Paths” plugin and then click the “Install plugin” button. Then click “Close”.

    Install the Points to Paths plugin.

    Install the Points to Paths plugin.

  2. Pace bus doesn’t provide the “shapes.txt” file so we’ll need to find a new GTFS package. Download the GTFS package provided by the Chicago Transit Authority, which has bus and rail service in Chicago and the surrounding municipalities.
  3. Load the CTA’s “shapes.txt” file into the table of contents (Layers Panel) by following steps 4 and 5 in the first section of this tutorial.  Note that this data includes both the bus routes and the train routes.

    QGIS load CTA bus and train stops

    Import CTA bus and train stops into QGIS

  4. Now let’s start the conversion process. Click on Plugins>Points to Paths. In the next dialog box choose the “shapes” layer as your “Input point layer”.
  5. Select “shape_id” as the field with which you want to “Point group field”. This tells the plugin how to distinguish one bus route from the next.
  6. Select “shape_pt_sequence” as the field with which you want to “Point order field”. This tells the plugin in what order the points should be connected to form the route’s line.
  7. Click “Browse” to give the converted output shapefile a name and a location with your computer’s file browser.
  8. Make sure all  of the options look like the one in this screenshot and then click “OK”. QGIS and the plugin will start working to piece together the points into lines and create a new shapefile from this work.

    These are the options you need to set to convert the CTA points (stops) to paths (routes).

    These are the options you need to set to convert the CTA points to paths (routes).

  9. You’ll know it’s finished when the hourglass or “waiting” cursor returns to a pointer, and when you see a question asking if you would like the resulting shapefile added to your table of contents (Layers Panel). Go ahead and choose “Yes”.

    QGIS: CTA bus and train points are converted to paths (routes)

    The CTA bus and train points, provided in a GTFS package, have been converted to paths (routes/lines).

  10. Now follow steps 1-4 from Section 3 to convert the routes/lines data to a shapefile or GeoJSON file.**

Notes

* As of this writing, the schedules in Pace’s GTFS package are accurate as of January 18, 2016. It appears their download link always points to the latest version. Transit schedules typically change several times each year. Pace says, “Only one package is posted at any given time, typically representing Pace service from now until a couple of months in the future. Use the Calendar table to see on which days and dates service in the Trips table are effective.”

** Choose GeoJSON if you want to show this data on a web map (like in Leaflet or the Google Maps API), or if you want to share the data on GitHub.

Obtaining Chicago Transit Authority geodata

October 12, 2010 / / 0 Comments

A reader asked where they could get Chicago Transit Authority (CTA) data I didn’t already have on the “Find GIS data” page. I only had shapefiles for train lines and stations. Now I’ve got bus routes and stops.

You can download General Transit Feed Specification (GTFS) data from the CTA’s Developer Center. It’s updated regularly when service changes.

Screenshot from ESRI ArcMap showing the unedited shapes.txt file loaded via Tools>Add XY Data. Shapes.txt is an 18 MB comma-delimited text file with thousands of points that can be grouped together with their shape_id.

The GTFS has major benefits over providing shapefiles to the public.

  1. It can be easily converted to the common shapefile format, or KML format.
  2. Google, the inventor of GTFS, has defined and documented it well; it is unencoded and plaintext. These attributes make it easy for programmers and hackers to manipulate it in many ways. (see also item 4)
  3. Google provides a service to the public on its website, an easy to use and robust transit planning service.
  4. The data is stored as plaintext CSV files.
  5. While an agency like CTA may have a geodata server on its intranet, it is less likely it has the addons that provide mapping and geodata services for the internet. A server like Web Mapping Service, or ArcIMS. These systems can be expensive to purchase and license. And we all know how the CTA seems to always be in a money crunch. While the CTA updates its GTFS data for publishing to Google Maps, the public can download it simultaneously to always have up-to-date information, providing the same geodata that ArcIMS or WMS would offer but for no additional cost.

I couldn’t have pulled off this conversion in 24 hours without the help of Steven Romalewski’s blog, Spatiality. He pointed me to the right ArcMap plugin in this post about converting the Metropolitan Transportation Authority’s GTFS data into shapefiles. I hope Steven doesn’t move to Chicago less my authority on GIS and transit be placed in check!

Make your own map of the CTA train routes and perform some kind of analysis – then share it with the rest of us!

Read more about my exercise in geodata conversion in the full post.
Continue reading

How to convert GTFS to GIS shapefiles and KML

October 12, 2010 / / 41 Comments

This tutorial will teach how you to convert any transit agency’s General Transit Feed Specification (GTFS) data into ESRI ArcGIS-compatible shapefiles (.shp), KML, or XML. This is simple to do because GTFS data is essentially a collection of CSV (comma separated values) text files (really, really large text files).

Note: I don’t know how to do the reverse, converting shapefiles or other geodata into GTFS data. I’m not sure if this is possible and I’m still investigating it. If you have tips, let me know.

Converting GTFS to GIS shapefiles

Instructions require the use of ArcGIS (Windows only) and a free plugin called ET GeoWizards GIS for any version of ArcGIS. I do not have instructions for Mac users at this time.

I wrote these instructions while converting the Chicago Transit Authority’s GTFS files into shapefiles based on a reader’s request. “Field names” are quoted and layer names are italicized.

  1. Download the GTFS data you want. Find data from agencies around the world (although not many from Europe) on GTFS Data Exchange.
  2. Import into ArcGIS the shapes.txt file using Tools>Add XY Data. Specify Y=lat and X=lon
  3. Using ET GeoWizards GIS tools, in the Convert tab, convert the points shapefile to polyline.
  4. Select the shapes layer in the wizard, then create a destination file. Click Next.
  5. Select the “shape_id” field
  6. Click the checkbox next to Order and select the field “shape_pt_sequence” and click Finish.
  7. Depending on the number of records (the CTA has 466,000 shapes), it may take a while.
  8. The new shapefile will be added to your Table of Contents and appear in your map.
  9. Import the trips.txt and routes.txt files. Inspect them for any NULL values in the “route_id” field. You will be using this field to join the routes and trips table. It may be a case that ArcGIS imported them incorrectly; the text files will show the correct data. If NULL values appear, follow steps 10 and 11 and continue. If not, follow steps 10 and 12 and continue. This happens because ArcGIS inspected some of the data and determined they were integers and ignored text. However, this is not the case.
  10. Export the text files as DBF files so that ArcGIS operates on them better. Then remove the text files from the Table of Contents.
  11. (Only if NULL values appear) Go into editing mode and fix the NULL values you noticed in step 9. You may have to make a new column with a more forgiving data type (string) and then copy the “route_id” column into the new column. Then continue to step 12.
  12. Join routes and trips based on the field “route_id” – export as trips_routes.dbf
  13. Add a new column to shapes.shp called “shape_id2”, with data type double 18, 11. This is so we can perform step 14. Use the field calculator to copy the values from “shape_id” (also known as ET_ID) to “shape_id2”
  14. Join routes_trips with shapes into routes_poly based on the field “shape_id” (and “shape_id2”)
  15. Dissolve routes_poly on “route_id.” Make sure all selections are cleared. Use statistics/summary fields: “route_long,” “route_url.” Save as routes_diss.shp
  16. Inspect the new shapefile to ensure it was created correctly. You may notice that some bus routes don’t have names. Since these routes are well documented on the CTA website, I’m not going to fill in their names.

Click on the screenshot to see various steps in the tutorials.

Converting GTFS to KML

After you have it in shapefile form, converting to KML is easy – follow these instructions for using QGIS. Or if you want to skip the shapefile-creation process (quite involved!), you can use KMLWriter, a Python script. Also, I think the latest version of ArcGIS has built-in KML exporting.

Converting GTFS to XML

If you want to convert the GTFS data (which are essentially comma-separated value – CSV – files) to XML, that’s easier and you can avoid using GIS programs.

  • First try Mr. Data Converter (very user friendly).
  • If that doesn’t work, try this website form on Creativyst. I tested it by converting the CTA’s smallest GTFS table, frequencies.txt, and it worked properly. However, it has a data size limit. (User friendly.)
  • Next try csv2xml, a command line tool. (Not user friendly.)
  • You can also use Microsoft Excel, but read these tips and caveats first. (I haven’t found a Microsoft application I like or think is user friendly.)