Robert A. Collins of Florida Atlantic University, with the counsel of Roger C. Schlobin of Purdue University and then aspiring author Timothy R. Sullivan, first conceived the idea of the International Conference on the Fantastic in the Arts in 1979 after attending Marshall B. Tymn's 4th Conference on Teaching Science Fiction in Boca Raton, Florida. Collins had long bemoaned the hostility with which readers and students of fantasy literature were greeted by scholars, and Schlobin had recently been rebuffed by members of the SFRA for suggesting they add the word "fantasy" to the description of what their organization researched. Collins attempted to launch a conference focused on the study of all literature of the fantastic, including science fiction works as well as those in other "genre" classifications. Together, Collins, Schlobin, and Sullivan opened requirements for submissions, using Eric Rabkin's definition of the Fantastic as any literature in one hundred eighty degree opposition to realism.

With a gift of operating funds provided by Margaret Gaines Swann, mother of the late FAU professor and fantasy author Dr. Thomas Burnett Swann, Collins organized the first ICFA on an experimental basis on the campus of FAU. The first meeting, in March of 1980, was so successful that it inaugurated an annual celebration of the fantastic, with the members of FAU's English and foreign language departments refereeing papers and editing annual proceedings. In 1982, Collins and several interested parties at the 3rd ICFA meeting formed the International Association for the Fantastic in the Arts, electing Jules Zanger President and Roger Schlobin Vice President.

Collins would continue as Conference Organizer until after the 5th meeting in 1984, when FAU and Mrs. Swann withdrew funding. After relentless searching for a new home for ICFA, now-President Schlobin finally accepted Lamar University's offer to underwrite the 6th meeting and relocated to Beaumont, Texas, in 1985. At this time, undergoing severe personal financial trouble, Collins declared himself unable to attend, but Schlobin generously responded with the establishment of the Robert A. Collins Service Award, at that time granting to faithful servants of the ICFA, as occasion dictated, a lifetime membership in IAFA with registration fees paid at all future ICFAs. Collins was the first recipient at the Beaumont meeting in 1985, Schlobin the second in 1986. By this time Marshall B. Tymn had taken the reins as third IAFA President, moving the 1986 and 1987 meetings to Houston.

The Houston years, underwritten almost entirely by Hap Henriksen's Carnegie-Mellon grant to create a Museum of Fantasy and Science Fiction, were abundant ones, and when the grant expired (with no museum existent), Tymn moved the ICFA to Dania, Florida, and the Ft. Lauderdale Airport Hilton, recruiting the joint sponsorship of Broward Community College and FAU.

In 1988, IAFA began publishing the Journal of the Fantastic in the Arts, an interdisciplinary quarterly founded and edited by then IAFA Vice President Carl Yoke. In 1989 (after ten years of appointments made by committees), IAFA held its first general election for positions on the Executive Board, resulting in Donald E. Palumbo and C.W. Sullivan III taking the helm as President and Vice President, respectively. At the 1991 meeting, Sullivan was then elected President, with Nicholas Ruddick as Vice President.

First Sullivan and then William A. Senior, elected IAFA President in 1994, worked for several years toward saving the ICFA from near-bankruptcy, making it not only solvent but smoothly self-supporting.

Meanwhile, Donald E. Morse (deserving recipient of the Collins Award in 1990) had all along exerted heroic efforts to organize each meeting since ICFA had first moved from Collins' home town of Boca Raton back in 1985. In 1991, Morse researched options for relocating but eventually accepted a new contract with the Ft. Lauderdale Airport Hilton, where the ICFA remained until it moved in 2008 to the Marriott Orlando Airport Hotel.

Following the Presidency of William Senior from 1994-98, the IAFA elected Len Hatfield President and also created the position of Second Vice President on the Executive Board, electing Veronica Hollinger First VP and Joe Andriano Second VP. In 2002, when Hatfield's term was to end, the Board faced some difficulty filling his position. Past Presidents Sullivan and Senior then agreed to serve as President in alternating years and did so until Mike Levy was elected in 2004, with Farah Mendlesohn and Robin Reid as First and Second VPs, respectively. In 2007, Mendlesohn was elected President, with Christine Mains as First VP, joining Robin Reid already serving as Second VP.

In 2009, Sydney Duncan was elected to succeed Robin Reid as 2nd VP, and Bill Clemente was re-elected Treasurer. In early 2010, the membership elected Jim Casey President, Sherryl Vint First Vice President, and (for what is newly an elected position) Crystal Black Public Information Officer.

Coming up on its thirty-second meeting, the International Conference on the Fantastic in the Arts is still one of the most diverse, energetic, provocative, and addictive interdisciplinary gatherings in the world. While the ICFA itself is real, it still reveres Rabkin's ideal for art: a one hundred eighty degree opposition to armchair realism. It does this additionally through its online newsletter and the Journal of the Fantastic in the Arts but primarily through its magical, other-worldly, annual meetings.