They Can't All Be Zingers includes 16 digitally remastered songs that span their entire career, including a previously unreleased and extended version of "Shake Hands With Beef", the Tom Waits collaboration "Coattails of a Dead Man", and "Mary the Ice Cube" from the 2003 DVD/EP Animals Should Not Try to Act Like People.
The Best Buy version of the album contains a bonus CD with four rare live tracks ("Those Damned Blue Collar Tweekers," "Bob," "My Name Is Mud," & "Jerry Was a Race Car Driver") recorded at Woodstock '94, including the infamous version of "My Name Is Mud" where they were pelted with mud by the audience (in which Claypool responds by informing them of their "small and insignificant genitalia").
The album's title comes from a line from the stand-up act of comedian Neil Hamburger. The CD is packaged with an outer cellophane cover meant to represent a package of individually wrapped single slices of processed cheese. The artwork for the inner booklet and the CD looks like cheese itself. The wedge of cheese pictured on the sleeve has the shape of the head in the frying pan from Frizzle Fry emerging from it.