Skip to main content

Archives

A sample text widget

Etiam pulvinar consectetur dolor sed malesuada. Ut convallis euismod dolor nec pretium. Nunc ut tristique massa.

Nam sodales mi vitae dolor ullamcorper et vulputate enim accumsan. Morbi orci magna, tincidunt vitae molestie nec, molestie at mi. Nulla nulla lorem, suscipit in posuere in, interdum non magna.

[QWIPU] Change in compression format for QWI download files

Print Friendly, PDF & Email

Given last year's addition of two additional QWI tabulation types (by age and education, and by race and ethnicity), the average size of a single release of QWI tabulations has increased from around 100GB to over 300GB. As a consequence, we have reduced the window of previous releases that we keep available on our download servers, and we will be transitioning from GZIP to a stronger compression algorithm (BZIP2). For nearly all of our download clients, this should not change anything, however the technical details may matter for some.

  • Previously, we had kept up to two years worth of prior releases. We are now reducing that window to one year of releases (four consecutive releases). Note that each release of the QWI public use data has the complete time series, with the latest release containing the most up-to-date information incorporated into the time series, so this only affects the ability to track specific versions of such releases. At this point, all previous releases are still available on the SSG compute servers.
  • We have traditionally kept the same compression algorithm used by the Census Bureau (GZIP). However, BZIP2 achieves file sizes about 30% smaller. The BZIP2 algorithm should be incorporated into most modern compression software (WinZIP, etc.) and is available in free Unix/Linux versions as well. Please let us know if this causes any problems by writing us.