Smith attended Maury High School and played at the University of Maryland. In the 1995 NBA Draft, Smith was selected by the Golden State Warriors as the first overall pick, before fellow power forwards Kevin Garnett, Antonio McDyess and Rasheed Wallace, as well as guard Jerry Stackhouse. Smith was named to the NBA All-Rookie First Team for the 1995–96 season and finished second in the Rookie of the Year voting to Damon Stoudamire.

Joe Smith would play two-and-a-half years for the Warriors before being traded to the Philadelphia 76ers midway through the 1997–98 season. Smith's statistics declined immediately after the trade, and have never fully recovered.

 Minnesota Timberwolves (1998–2003)

Despite a drop in production, Smith was seen still as a hot commodity in the free agency blitz^{[citation needed]} that followed the strike in late 1998. In what seemed at the time like a bizarre move, Smith signed for very little money with the Minnesota Timberwolves. For the next two years, Smith played very productively at small forward alongside All-Star Kevin Garnett.

 Timberwolves salary cap scandal

Following the 1999–2000 season, it was discovered that Smith was involved in a salary cap-tampering scandal involving Timberwolves executive Kevin McHale. Smith was allegedly promised a future multi-million dollar deal if he signed with the team for below market value, allowing the team to make some additional player moves in the short term. The league later found out about this violation and voided the last year of the contract, also severely punishing the Timberwolves by taking away five first-round draft picks (though two of the picks were ultimately returned) and fining the team $3.5 million.The move hurt the Timberwolves in the long run, as while the team still found success, the lack of draft picks set the team up for their eventual failure in the mid-2000s.

Detroit Pistons (2000–2001)

Joe Smith was released by the Timberwolves and signed with the Detroit Pistons for the 2000–2001 season as a backup. Smith produced very good numbers for the Pistons and, at the end of the season, he re-signed with the Timberwolves where he played for two more seasons.

Smith (left) driving against the Warriors' Chris Webber (#4) with the Bulls in 2008

NBA journeyman (2003-present)

For the next three seasons Smith played for the Milwaukee Bucks. Prior to the 2006–07 season, Smith moved on to the Nuggets, where he only played in 11 games before being traded, along with Andre Miller, back to the 76ers for Allen Iverson. Despite being considered an add-in on the deal, Smith played a decent part in the 76ers spectacular second half to the season as they just barely missed the playoffs.

For the 2007–08 season, Smith signed with the Chicago Bulls. Smith played very well for the Bulls, but the rest of the team struggled mightily. Smith was traded in a three team deal at the trade deadline to the Cleveland Cavaliers.

The Cavs traded Smith to the Oklahoma City Thunder on August 13, 2008, as part of a three-team deal that also included the Milwaukee Bucks.

The Thunder traded Smith to the New Orleans Hornets along with Chris Wilcox and draft rights to Devon Hardin on February 17, 2009 for Tyson Chandler but on February 18, 2009 the trade was rescinded after Tyson Chandler failed a physical with Oklahoma City.

On March 1, 2009 The Thunder bought out the remainder of Smith's contract and released him. Two nights later, he agreed to terms to return to Cleveland and rejoin the Cavaliers.

On August 25, 2009, he signed a one year contract with the Atlanta Hawks.