Disaggregation

 

Pave the World

I swear I'm in fourth gear,
My rollers down, trail of smears.
Asphalt, aggregate,
No more trees, pavement.

If you don’t like aggregate, then how do you feel about disaggregation.

I am returning to the original purpose of this blog, which was to keep from repeating my self to colleagues, and instead referring them to a link to a blog post. So while the subject probably does not interest the vast majority of you, here it goes.

Disaggregation

Disaggregation is merely the opposite of aggregation. Matrices might be aggregated; in which case they can of course be disaggregated. There are also cells in the matrix which might be required to be aggregated, or otherwise identified, for example cells that pass through the area to be “disaggregated.” A knowledge of which cells pass through the area to be disaggregated, for example a network and/or its routing table which identifies which cells of the matrix use links of the network that are located in the area being disaggregated, then also needs to be provided. The application of disaggregation factors, without identifying which links in the disaggregated network should be used, is called matrix expansion. The filtering of cells in the matrix that use links in the disaggregation area is variously called subarea extraction, windowing, or select link analysis. For purposes of this discussion, disaggregation is taken to mean that only occurring without a network and includes only cells that travel to, from, or within the study area which is being disaggregated. This is also known as I-I, I-E and E-I trips, where I are the internal cells to be disaggregated and E are the external cells that are not to be disaggregated. Cells that are E-E, which begin outside of the area to be disaggregated, end outside of the area to be disaggregated, but also use links within the area to be disaggregated, i.e. E-E Pass-through, use a different method than matrix expansion and its disaggregation factors.

Disaggregation, i.e. matrix expansion, requires the identification of how the matrix was aggregated in the first place. This is not always known, or even knowable. The matrix might not have been aggregated. For example, the Federal Highway Administration’s, FHWA’s, Freight Analysis Framework, FAF, is developed from the Commodity Flow Survey, CFS and methods applied to shipments that are outside the scope of the Commodity Flow Survey. Those CFS Out Of Scope, OOS, flows, were not aggregated. The Commodity Flow Survey flows were aggregated by applying expansion factors to surveys. However the surveys themselves are not disclosed to protect the confidentiality of the survey respondents. Those surveys also do not represent 100% of the flows. Their expansion factors only apply to the groupings of the surveys which are statically significant. There are no expansion factors for grouping of cells which were not statistically significant. The absence of aggregation factors  means that it is not possible to develop the guaranteed disaggregation factors. However that does not mean that it is not possible to develop most probable disaggregation factors.

The concept of the most probable is common in transportation planning.

·        Frataring, often known as IPF, Iterative Proportional Fitting.

·        the Gravity Model.

·        the Logit Mode Split Model; and

·        Origin Destination Matrix Estimation, ODME, etc.

are all based on finding the most probable trip table. Disaggregation itself is an intermediate, meso, state between individual, micro, states and the aggregated, macro, state. The desired matrix is an intermediate, thus the mathematical concept of the most probable, the mesostate which contains the greatest number of microstates, can be used. However using the most probable disaggregation requires honoring the concept of the most probable disaggregation.

Dimensions Being Disaggregated.

Theoretically any of the indices, dimensions, primary key fields of the matrix, can be used in disaggregation. For example, the FAF has four primary key fields:  the origin, the destination, the commodity, and the mode. While it is possible to disaggregate commodities in the FAF, which are currently reported as Standard Classification of Transported Goods at the two-digit level, SCTG2, by a greater number of SCTG digits, this is not often done (using a different Commodity classification system would be CROSSWALKING, not DISAGGREGATING). While the modes reported in the FAF can be disaggregated (e.g. Water can be disaggregated into Inland Barges, Great Lakers, and Deep Sea vessels) the most common application of disaggregation is for the origins and destinations, geography, from the reported states and regions to smaller zones, for example to county-equivalents in those states and regions.

Correlation is not causation

Disaggregation can be on any characteristics of the dimensions of the matrix. For example, a region with two counties can be disaggregated 50% to each county for both origins and destinations. However it is not expected that this disaggregation factor will be correlated with the number of those smaller zones. For example, it might be more likely that origins will be correlated with employment and that destinations will be correlated with population. If the two counties had respectively 70% and 30% of the employment and 60% and 40% of the population, those characteristics might have been used as disaggregation factors instead. However those factors do not consider the commodity or mode. It may not be possible to develop or obtain factors that are CAUSING the flow. It is more likely that information showing CORRELATED factors. In this case, care should be taken that the correlation makes sense and is not a spurious correlation.

Perspective Matters

The nature of the matrix being disaggregated may have information that can be used in creating disaggregation factors. For example, in the transportation of freight, there are three functions: shipper, receiver, and carrier. It is possible for two or more of those functions to be combined: e.g. Dole may carry goods that it receives in trucks which it owns. Wal-Mart may carry goods in trucks which it owns, and additionally serve as both the shipper and the receiver, and that flow may not even be reported if it is an internal business operation. The source of the matrix may provide information on the dimensions and the disaggregation factors to be used.

As noted, the FAF is derived from the Commodity Flow Survey of shippers. The dimension is an origin, but the Domestic Origin is the origin of the shipper for domestic trade, while the Foreign Origin is the origin of the shipper for import trade. The Domestic Origin for imports, and the Domestic Destination for exports, is more properly a Port Of Entry/Exit, POE, which is a carrier transfer point at which no economic activity of the shipper or receiver is likely to occur.

Because the flows are as reported by the shipper, the mode used by the carrier may not be known, only the rate that was charged by the carrier. Thus a mode in the FAF is Air and Truck where the shipper pays a rate that is typical of air, but the carrier might choose operationally to use a truck. Because the flow is as reported by the shipper, the commodity is always known which is why the SCTG2 system, which does not include Freight All Kind, FAK, is used, while the carrier may not know the commodity and use the Standard Transportation Classification Code commodity system which does include FAK. Similarly the FAF does not include the packaging of the commodity in containers, while containers may be an important reporting by carriers.

In carrier focused matrices, the origins may include both carrier and shipper origins and the destinations may include both carrier and receiver destinations. A source of economic activity from an origin may only be pertinent for shipper-receiver matrices, such as the FAF, not carrier matrices such as Transearch. The first origin and the last destination may also reflect the economic activity of a customer, or a carrier operations while intermediate stops as reported in modal databases, are almost always carrier operations/activity.

Disaggregation Factors should be unique.

The matrix is probably not symmetric and has unique flows in each cell, and thus the disaggregation factors should also be expected to be unique. This is the ideal. It may be that disaggregation factors can be created for several dimensions, but not for all dimensions. In which case something is better than nothing. For example, import trucks and trains are reported at Border Crossing Port of Entries. The commodity contents of those trucks and trains are not reported and the export flows are not reported. It can be assumed that the relative share for vehicles is the same as the relative share for the commodities in those vehicles, and the relative share for imports is the same as the relative share for exports. While this is hardly unique, it is preferable to making no assumptions at all.

Disaggregation factors should sum to 100%.

Disaggregation factors must sum to 100% for each dimension for each cell that is being disaggregated. Thus if regions are being disaggregated to counties, the disaggregation factors for those counties should sum to 100% for each region. This means that the absolute number of shipments from each county is not needed to create disaggregation factors, only the relative number of shipments, activities, for each county in that region.

The number of disaggregations, mesostates, should be kept as small possible.

The most probable disaggregation factors depend on having as many microstates as possible. The number of microstates will not change. Increasing the number of disaggregation factors for each cell increases the number of mesostates but decreases the number of microstates in each mesostate.

The most probable is not a  guarantee. For example in the game of craps, the most probable roll over a very large number of rolls is a seven. On the next roll the total may be a two, but this does not change the fact that seven was still the next most probable roll. To ensure that the most probable event has meaning, the number of events, i.e. the number of microstates in a mesostate, should be kept as high as possible which means that the number of mesostates should be as low as possible.

Activities should correlate with a disaggregation factor.

Economic activity may apply to all goods transported from a shipper to a receiver. However only those economic activities that also use a mode should be related to any disaggregation of those modal flows. Information is available on which counties are not served by active rail tracks or navigable water. This does not mean that they are served by rail or water carriers, only that if the tracks or navigable water do not exist, then service by carriers could not be offered. Such information is not available for other modes and as a default all economic activity may be considered in creating disaggregation factors. While there is no information on trucks, it is assumed that trucks serve all economic activities.

Disaggregation factors in the tool that I developed

Domestic economic activity

For the domestic trade type, the origin is the domestic origin, and the destination is the domestic destination. The disaggregation factors should also be by commodity and by mode. As noted above, the Oak Ridge National Laboratory’s Center of Transportation Analysis has identified counties that are NOT servicable by rail or water. And this information can be used in developing relative disaggregation factors. The economic activity information is not generally available by commodity but is available by employment in an economic sector, which is generally the industry by the three-digit North American Industrial Classification System, NAICS3. Economic Input - Output models have Make and Use tables which relate economic sectors with commodities. The Bureau of Economic Analysis maintains Make and Use tables and crosswalks can be developed between its commodity and economic sectors,  and NAICS3 industries and SCTG2 commodities. The BEA Make and Use tables are for average percentages of value by economic sector. These averages are just that and a firm may be more or less productive than the average. Additionally the share of employment may not equate to the share of economic value. And the share of the economic value may not equate to the share of tonnages. However if these are assumed, it is possible to develop the economic activity by commodity for origin (shipper/Make) and destination ( receiver/Use.)

Import carrier activity

The economic activity of imports to the domestic destination is the same as the domestic economic activity above. The domestic origin for imports is a Port of Entry. For truck and rail, the Bureau of Transportation Statistics, BTS, Transborder flows, can identify the relative share of carrier activity at each truck and train border crossing in a FAF region. For water, the US Army Corps of Engineers Waterborne Commerce Statistics Center reports import tonnages by commodity. It uses the Public Group commodity classification codes, but those can be crosswalked to SCTG2 commodities. The relative share by water Port POE for each commodity for each FAF region can be calculated from this information. There is no information on the domestic mode which is used at these POEs. For all other foreign modes there is no information that can be used to develop disaggregation factors, so the POE is merely relabeled with its foreign mode and domestic region.

 Export carrier activity

The economic activity of exports from the domestic origin is the same as the domestic economic activity above. The domestic destination for exports however is a Port of Exit, POE. For truck and rail, the Bureau of Transportation Statistics, BTS, Transborder flows, can identify the relative share of carrier import activity at each truck and train border crossing and it is assumed that carrier export activity has the same share. For water, the US Army Corps of Engineers Waterborne Commerce Statistics Center, reports export tonnages by commodity by port waterway. It uses the Public Group commodity classification codes, but these can be crosswalked to SCTG2 commodities as used by the FAF. The relative share by water Port POE for each commodity for each region can be calculated from this information. There is no information on the domestic mode that is used at these POEs. For all other foreign modes there is no information that can be used to develop disaggregation factors, so the POE is merely relabeled with its foreign mode and domestic region.