![]() ![]() The chances that they will write Java, C or Python code to work with the data are slim. Making data available via the latest XML-WSDL-web-service frameworks may fit with software engineering best practices but these data are unlikely to be useful to biologists, environmental consultants, geologists, hospital administrators, petroleum engineers, physicists or anyone else without a computer science degree.Įxpecting these people to have access to computer staff who can help them is often a very poor assumption. Sometimes this means doing things in a less than cutting edge manner. It is up to the data managers to make sure that data are made available in structures and formats that help the ultimate users. A tremendous amount of time and effort is spent reformatting data for use with specific analysis tools and an equally tremendous amount of subtlety and detail is lost with each reformatting. In order for science to inform public policy, the process of working with scientific data must be made easier. To our way of thinking, scientific data management should be about meeting the needs of the data consumers - scientists, policy makers and engaged members of the public. Any data consumer wishing to generate a synoptic view of the data - all stations for a particular year - must reorganize the data in order to generate the maps or other broad scale representations they desire Samples taken at one location in different years have different timestamps but the same “Station ID”. Data are typically collected and organized by location through the use of a unique “Station ID”. In the world of environmental science, the reverse scenario is often true. After opening the file for their location of interest they would simply read all of the data. Clearly the “time-series user” would prefer the data be organized as “time-series by location”. A time-series of 1,000 points may require processing of a Terabyte of data. For these users, the snapshot world view will be an excellent fit.īut what about someone who is interested in looking at a time series representation of the daily weather or monthly climate at a particular location? To assemble the data for this representation, our “time-series user” must open and read each (potentially multi-Gigabyte) snapshot in order to extract a single value at their location of interest. There will be some who are interested in generating maps of weather or climate at some future time or date: the “map users”. This series of snapshots is the world view of these data producers.ĭata consumers come in many varieties of course. The output of these models, even when stored as multi-dimensional NetCDF files, is typically organized as a series of snapshots at specific times. ![]() The organizing principle for data input and output files is xy-region by time point.Ĭlimate models work the same way, calculating the state of the global climate one time point at a time. These fields are used as input to forecasting models that compare the most recent field with earlier fields and calculate a forecast of the state of the atmosphere at specific times in the future. Weather data is collected every hour and is fed into data ingest models that create as output synoptic fields - descriptions of the state of the atmosphere at a specific time but on a broad spatial scale. The same competing needs exist in the world of scientific data management where producers of data and consumers of data often operate in very different worlds with very different sets of tools.Īlthough examples could be drawn from any field of science, climate and environmental data provide some superb examples that highlight the different world views of data producers and data consumers. In the marketplace, the needs of producers and consumers are often at odds: producers want higher prices, consumers lower ones producers want easy assembly, consumers easy dis-assembly producers want flexibility and rapid prototyping, consumers reliability and long-term support. ![]()
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |