The Jacob K. Javits Federal Building was built in 1967 for the US Federal Government. It is currently home to the U.S. Immigration and Naturalization Service as well as a number of other government agencies including the New York division of the FBI. It has been described as the "modern Ellis Island" and is synonymous with impenetrable bureaucracy and overbearing security. Since 9/11, the building has also gained a controversial reputation for unexplained detentions and on-the-spot deportations.

Along with the adjacent Customs Courthouse, The Federal Office Building is the only completed part of an original 1962 plan for a Civic Center that envisioned a much more ambitious redevelopment of the whole area north of City Hall in New York. The plan centered on the construction of three separate towers that would house office space for Municipal and Federal Government. Inspired by Rockefeller Center, the plan included a sunken mall with ice skating rinks in an extensive park stretching four blocks, as well as a network of concourses with retail outlets and connections to transportation hubs underground. The plan appears to have been scrapped at local government level due to the prioritizing of traffic flow through the area in the approach to the Brooklyn Bridge. The Federal Building was the only part of the scheme that was built, as its remit fell outside of the city's authority.

The voice-over in the film is taken from a statement issued in November 1964 by the architects, Edward Durell Stone and David L. Eggers at the point of the first major redraft of the original grand plan for the area. Their idealistic description, in its desire for openness, is clearly at odds with the ultimate outcome of the plan, as well as with the bureaucratic and political blockage represented by Federal Plaza in its current incarnation:

"...The architect's struggle to create order and beauty in our cities is in large part a struggle to create open space. Space, that is, in which to resolve our idealism; space to speak eloquently for the dignity of the people; space to humanize the present and award to the future. Space, in short, that will contribute to the long-term economic and social well being of the community. The architects are confident that with this spacious concept, they have offered New Yorkers a truly appropriate expression of their role as citizens of the headquarters city of the world."

The title of the film 'Rise and Fall' is a play on this theme of architectural expediency and compromised planning decisions - as well as a punning description of the camera action of the central panning shot in the piece.