James Kuiper, A. Ayers, R. Johnson, M. Tolbert-Smith
Presented at the Third International Conference/Workshop on Integrating GIS and Environmental Modeling, Santa Fe, New Mexico, USA, January 21-25, 1996
Sponsored by the National Center for Geographic Information and Analysis (NCGIA)
ABSTRACT
A common problem encountered in Geographic Information System (GIS) modeling is the exchange of data between different software packages to best utilize the unique features of each package. This paper describes a project to integrate two systems through efficient data exchange. The first is a widely used GIS based on a relational data model. This system has a broad set of data input, processing, and output capabilities, but lacks three-dimensional (3-D) visualization and certain modeling functions. The second system is a specialized object-oriented package designed for 3-D visualization and modeling. Although this second system is useful for subsurface modeling and hazardous waste site characterization, it does not provide many of the capabilities of a complete GIS. The system-integration project resulted in an easy-to-use program to transfer information between the systems, making many of the more complex conversion issues transparent to the user. The strengths of both systems are accessible, allowing the scientist more time to focus on analysis. This paper details the capabilities of the two systems, explains the technical issues associated with data exchange and how they were solved, and outlines an example analysis project that used the integrated systems.
INTRODUCTION
Scientists in the Environmental Assessment Division (EAD) of Argonne National Laboratory are involved in a wide variety of projects and use many different software packages to conduct research. Although the variety of software packages used expands their capabilities to analyze and model information, it also can result in bottlenecks when data cannot be easily transferred from one software package to another. This situation is neither new nor unusual. Both individual software developers (at the application level) and organizations for standards development (at the industry level) are addressing this problem.
Two software packages frequently used by EAD are ARC/INFO(TM) (Environmental Systems Research Institute, ESRI Inc.), a Geographic Information System (GIS); and SiteView(TM) (ConSolve Inc.), a three-dimensional (3-D) visualization and analysis tool. Although both are designed for spatial data analysis with overlapping functionality, they each have unique capabilities. In the case study of a Department of Energy (DOE) project presented here, it is clear that both systems provided essential capabilities not available in the other. The case study involved a site at a DOE facility that was historically used to dispose of hazardous waste.
The two systems had only one common exchange format for unattributed line and polygon data: the AutoCAD(TM) (Autodesk Inc.) Drawing Exchange Format (DXF) (Autodesk 1993). Exchange of other data is limited to column-formatted text files. As shown by the case study, efficient data exchange allowed these two tools to be integrated for the project, taking advantage of the unique capabilities of each.
BACKGROUND
ARC/INFO is an extensive GIS with vector and raster capabilities including import, georeferencing, editing, analysis, and output. The system provides the capability to generate 3-D surface views, but it is not designed to model or display subsurface information. For the DOE case study, general geographic data for base maps of the study were available in ARC/INFO. The raster analysis module was used for non-intrusive geophysical survey data, including input, georeferencing, interpolation, and processing to produce imagery. It was also used to input and georeference scanned aerial photographs of the study area.
SiteView is more limited and lacks several of the major components of a GIS, but it is specifically designed for 3-D visualization and analysis of surface and subsurface site characterization data. It can easily import tabular data and point graphics, but for more complex spatial features it is limited to the DXF file format or manual input by the user. Data imported with the DXF format is for display only, and cannot be attributed beyond its graphics-related information. SiteView has an easy-to-use graphical user interface, and is much less expensive than ARC/INFO. In the DOE case study, SiteView was used to store and visualize subsurface data.
The difference in data models was the most significant obstacle to be overcome in integrating the two systems. ARC/INFO has a topological, feature-based structure for graphics and a relational database for tabular attributes. A linear item, such as a road, is given a line feature type. Attributes describing roads are stored in a relational database and attached to the lines by a system-maintained link. The line feature type maintains topology so line connections are known and can be used for analysis. Land-use areas would be stored with a polygon feature type in a separate data set. Polygon topology allows adjacent polygons and their shared boundary to be identified. In general, the feature type structure is designed for analysis, but relationships between data sets, such as selecting roads within a land-use polygon, must be calculated by the system or determined by the user in some manner. Although feature types such as points, lines, or polygons are designed into the system, the data set name and design of tabular attribute fields are user defined.
In contrast, SiteView uses an object-oriented data model to store information. The set of objects, such as well points or soil polygons is pre-defined, cannot easily be modified by the user, and is designed specifically for site characterization data. Land-use areas in this data model are stored as land-use objects containing both polygon coordinates and tabular attributes. The available attributes are pre-determined by the object's structure. Data for sampling wells are stored as well objects with both coordinates and attributes. Data for samples from the wells are stored as sample objects. The set of objects can be organized and used in more complex ways than a relational data model easily permits. Samples, when linked to a well, inherit the geographic location of the well. A set of wells can be associated with a land-use area. Once this is done, the set of samples can be identified based on land use using links inherent in the data. SiteView does not maintain topological relationships. For example, no function exists to determine if two polygons are adjacent. The line between them is duplicated rather than shared and may not necessarily be coincident.
Both systems and data models provided useful and unique capabilities, but the significant differences and lack of an efficient method of data exchange limited their combined use.
IMPLEMENTATION
A project to improve data exchange between the two systems was funded by the DOE. The work was designed to focus on some immediate needs of DOE environmental restoration projects, to provide generally applicable extensions to SiteView, and to result in code that could be useful for other similar data transfer applications.
The level of integration was limited to data translation using an intermediate data file, primarily because the systems differ significantly in data models as well as representation of more complex graphic elements, such as polygons. The method of integration also addressed proprietary issues associated with both software packages and transfer of data between separate users or systems.
Design of the Intermediate File Format
In early stages of the project, many established file formats were examined. In particular, a design based on the Spatial Data Transfer Standard (SDTS/FIPS-173) (NIST 1992) was considered. Import and export of SDTS files was newly supported by ARC/INFO (ESRI 1995a) and the standard provided detailed guidance for file formatting and other issues. SDTS was designed to be an all-encompassing standard and had many elements beyond the scope or needs of the project. Although it was an excellent source for design concepts, it did not offer an ideal solution.
ArcView(TM) (ESRI, Inc.) Shapefiles were also considered. Import and export of this format was also supported in ARC/INFO. Shapefiles are binary and store graphics and attributes in a nontopological format (ESRI 1995b). This is an example of a software development organization designing a format optimized for its product. SiteView lacks a similar transfer format geared to its data model. It would be useful for SiteView to support the Shapefile format, but a format optimized for SiteView would provide more options for data exchange from a variety of sources.
The approach selected was to design and implement an ASCII file format that was uniquely adapted to SiteView and its data model. This approach allowed the format to be easily adapted to other uses, since ASCII files can be easily created or modified by users. The ASCII file consists of an informational header, a structure definition, and data records (Figure 1). The structure definition identifies the object type, geometric type, and the data attributes. The data section begins by identifying the object type and the data elements follow. Each data element begins with a tab separated list of attribute values followed by one or more coordinates, depending on the object type. Multiple structure definition and data sections are permitted in the file. A detailed file specification was documented by ConSolve (ConSolve 1995).
***SLF_FILE***
/work/doe/translate/roads83.slf
Mon Dec 4 08:50:35 1995
SiteLink v1.0
This data set contains road lines for the DOE study area.
;;;
Road
Polyline
Name|30|String|-1
PavementType|10|String|-1
;;
;;;
Road|93
1|paved|
3.95822218671853e+05| 7.80876320926992e+04| 0.00000000000000e+00
3.95823002518686e+05| 7.80952362633142e+04| 0.00000000000000e+00
3.95823860547643e+05| 7.81000000000000e+04| 0.00000000000000e+00
;
93|dirt|
3.95606244063250e+05| 7.81000000000000e+04| 0.00000000000000e+00
3.95612311693291e+05| 7.80878534624764e+04| 0.00000000000000e+00
3.95634650550764e+05| 7.80479487511311e+04| 0.00000000000000e+00
;
;;
;;;
Figure 1. Example of the transfer file format. Tab characters are shown as "|" characters for legibility. Only the first and last data records are shown.
Approach to Coding
The programmers had several choices for coding the ARC/INFO translator. One option was to use Arc Macro Language (AML) to transfer the information between ARC/INFO and the file. This method had the advantage of being compatible with all ARC/INFO-supported platforms, but the disadvantages of slow execution speed and limited language capabilities.
The coding approach chosen was to use ARC/INFO software development libraries (ArcSDL(TM)) and C code. The object libraries allow FORTRAN or C code to use most of the low-level routines in ARC/INFO to access data in its internal format. The final result can be a standalone application. The main disadvantages with this approach are portability (because the libraries and compiler are sold separately for different platforms), and the requirement of an ARC/INFO license to run the program. The positive factors that led to choosing this option were the speed improvements of compiled code, the ability to do all processing within one program, direct access to all ARC/INFO internal data types, and more flexibility in the design process. Also, the code developed would be useful for other future integration projects. The Galaxy Application Environment(TM) (Visix Software, Inc.) was used for the interface. It is a cross-platform development tool with an efficient graphical user interface (GUI) builder. It is also used by ConSolve to develop their products.
The User Interface
Import of the transfer file was added to the SiteView "File" menu as a new option. On the ARC/INFO side of the translation, the program was coded as a standalone application, SiteLink, with a graphical user interface. Most of the translation work is done in this application.
One of the more difficult issues was how to efficiently map data sets and attribute data fields from ARC/INFO's user-defined approach to SiteView's predefined object classes. The method developed was to add the SiteView object dictionary as a resource to the SiteLink translator. This allowed ARC/INFO data to be mapped to SiteView objects at the time of export. For example, the process to create a transfer file for an ARC/INFO roads data set and import it to SiteView would be as follows:
In SiteLink, use the file browser to load the ARC/INFO roads coverage.
Open up the list of arc attribute table fields.
Choose a destination SiteView object class. The interface limits the user to object classes that can contain lines, in this case, the "Road" object type.
Choose SiteView attribute slots for the ARC/INFO attributes to be transferred. The user is limited to unused SiteView attribute slots with a proper data type. Here, the INFO item "TYPE" is mapped to the SiteView attribute slot "PavementType" (Figure 2a).
Save the data as a transfer file using the file menu.
In SiteView, use the "Import Transfer File" option in the "File" menu to load the contents of the file (Figure 2b).
(a) (b)
Figure 2. (a) Using the SiteLink translator to translate an ARC/INFO roads data set. The "Road" object type has been assigned, and the PavementType attribute slot is being assigned for the INFO attribute "TYPE." (b) Importing the roads SiteLink transfer file into SiteView.
Solutions for System Differences and Limitations
The translation program needed to provide useful functionality and appropriately preserve significant information in the data. This required deciding which data types to support and designing algorithms to overcome differences in geometric object representation. Not all ARC/INFO data types were included for translation. SiteView does not support raster data storage, so the ARC/INFO Grid format was excluded. 3-D surfaces can be generated in either system, and were not included. Annotation and nonspatial attribute tables were omitted because they could be transferred by existing means.
Differing polygon representations was the major geometric issue. ARC/INFO polygons have a list of lines that define a closed boundary. The attributes are linked to a point inside the polygon. Lines are shared between adjacent polygons and holes within polygons are recognized. In SiteView polygon objects, the label point is not necessary and the attributes are an integral part of the object. The full closed perimeter of each polygon is specified and lines between adjacent polygons are duplicated. Holes are not possible, and must be represented as additional polygons on top of another one. The SiteLink transfer file is modeled after the SiteView format. ARC/INFO polygon holes are treated as separate polygons.
Regions in ARC/INFO are composed of one or more areas enclosed by rings of arcs. The discontiguous areas are treated as one attributed unit, can contain holes, and can overlap with other regions. In SiteLink, regions are output as a set of polygons, each having the region attributes. Region holes are also output as polygons, but attributes are assigned as null values. The programming challenge was to separate holes from additional discontiguous areas, since this is only represented by the direction in which the vertices are stored. Clockwise storage indicates a discontiguous area and counter-clockwise storage indicates a hole. Direction is determined by calculating the area, assuming that vertices are stored in a clockwise direction. If the calculated area is negative, then the vertices are stored counter-clockwise and the polygon is a hole.
SiteView stores point data in three dimensional form, while ARC/INFO does not. In ARC/INFO, elevation is stored in an attribute field. The SiteLink translator allows SiteView Z-coordinate values to be populated from an ARC/INFO numeric attribute. This was implemented efficiently in the user interface by adding a Z field to the menu choices for mapping numeric items.
For the case where data need to be added to SiteView, but an appropriate existing object class did not exist, generic object classes were added. These classes, named after the type of feature, (point, line, or polygon) have a set of generic attribute fields. In these objects, the ARC/INFO data set name is automatically transferred to a SiteView attribute so that the data can be easily identified and grouped in SiteView.
CASE STUDY
The combined capabilities of the GIS system, translator, and 3-D visualization package were used to help study a DOE site with subsurface contamination. The site included an unknown number of waste disposal pits that had to be located and sampled. The site characterization objectives were to determine the number, locations, and sizes of the disposal pits, and to determine the nature and extent of any contamination emanating from the pits. The first phase of the work used non-intrusive geophysical techniques including magnetic gradiometry, electromagnetic surveying, and ground penetrating radar profiling to locate the pits. The second phase was an intrusive sampling program targeting soils beneath and adjacent to identified disposal pits.
ARC/INFO and SiteView software were used to develop most of the figures produced for the DOE work. In combination they provided the ability to input, process, analyze, model, and visualize a variety of spatial data derived from existing data sources and field data collection. With the SiteLink translator, data that resided only in ARC/INFO was efficiently translated to SiteView.
ARC/INFO provided support for data import, georeferencing, and surface data interpolation and display. A base map and 1962 aerial photograph of the study site is shown in Figure 3. Several of the main disposal pits are visible in the photograph, as well as smoke from open burning at a nearby conventional landfill. Figure 4 depicts the results of the magnetic gradiometer survey of the site. This survey method identifies variations in magnetic fields at the ground surface and the light and dark areas represent positive and negative magnetic fluctuations. It successfully identified the disposal pits.
SiteView was used for most of the intrusive survey data because of its ability to produce 3-D views of subsurface data, fence diagrams, and analysis of disposal pit volumes. In Figure 5a, a subsurface view of the study area is shown with the disposal pits and slant bores. The two disposal pits identified in Figure 4 are highlighted. On the surface, road lines and forest polygons translated from ARC/INFO are shown. A groundwater depth surface is shown in Figure 5b. This surface was created in SiteView using the wells and depth-to-groundwater data.
Figure 3. Base map produced in ARC/INFO with 1962 aerial photograph showing the study area.
Figure 4. Nonintrusive magnetic gradiometer survey data shown in ARC/INFO.
(a) (b)
Figure 5. (a) Subsurface view of the study area in SiteView showing disposal pits and slant bores. Surface map layers were translated from ARC/INFO. (b) Groundwater depth visualized in SiteView with translated base map layers superimposed on the surface.
CONCLUSIONS
The SiteLink translator provides a simple method for importing data to SiteView and insulates the researcher from some of the more complex and detailed translation issues. This enables more productive analysis, as is evident in the case study. Much of the detail involved in reformatting information and mapping it to the target system is simplified for, or transparent to, the user.
The SiteLink translator benefits ARC/INFO users by giving them new 3-D visualization capabilities and SiteView users by giving them streamlined access to data from a widely used GIS. The transfer file format was designed as a general purpose file for SiteView and can be easily implemented for exchange with other systems. For the ARC/INFO user, SiteLink can also be implemented for general purpose file export, especially for transfer to object-oriented systems or those having similar data representation designs.
ACKNOWLEDGMENTS
The submitted paper has been authored by a contractor of the U.S. Government under contract no. W-31-109-ENG-38. Accordingly, the U.S. Government retains a nonexclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S. Government purposes. Work supported by the U.S. Department of Energy, Office of Environmental Management, under contract W-31-109-ENG-38.
Autodesk, Inc. (1993) AutoCAD Release 12 - Advanced Tools, pp. 267-315.
ConSolve, Inc. (1995) SiteLink File Transfers, Wayland, MA.
National Institute for Standards and Technology (1992) FIPS Publication 173: Spatial Data Standard, U.S. Department of Commerce.
ESRI, Inc. (1995a) SDTS: Supporting the Spatial Data Transfer Standard in ARC/INFO, Environmental Systems Research Institute, Inc. (Available from URL http://www.esri.com/resources/papers/papers.html.)
ESRI, Inc. (1995b) ArcView: Shapefile Technical Description, Environmental Systems Research Institute, Inc. (Available from URL http://www.esri.com/resources/papers/papers.html.)
NIST: See National Institute for Standards and Technology
AUTHOR INFORMATION
James A. Kuiper: Biogeographer / GIS AnalystCo-Authors:
Andy Ayers and Robert Johnson
Argonne National Laboratory / Environmental Assessment Division
Marilyn Tolbert-Smith
Office of Environmental Management
U.S. Department of Energy
A black and white Postscript version of this paper (as submitted to the conference on 12/15/95) is available as an uncompressed file (5.7 Mb), a Sun compressed file (1.1 Mb) or a GNU zipped file (1 Mb).
DISCLAIMERS
DISCLAIMER OF LIABILITY: This document was prepared as an account of work sponsored by an agency of the U.S.Government. Neither the U.S. Government nor any agency thereof, nor the University of Chicago, nor any of their employees or officers, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights.
DISCLAIMER OF ENDORSEMENT: Reference to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise does not necessarily constitute or imply its endorsement, recommendation, or favoring by the U.S. Government or any agency thereof. The views and opinions of document authors do not necessarily state or reflect those of the U.S. Government or any agency thereof, Argonne National Laboratory, or the University of Chicago.
February 1, 1996