|
Deal is a rowdy and rollicking song from the Grateful Dead, written by Jerry Garcia and Robert Hunter. Always a crowd pleaser!
The story behind the AI generated video
(WARNING: READ NO FURTHER UNLESS YOU WANT THE DETAILS).
Note: The terms AI and chatgpt are used interchangeably below.
Back in December, when I read that the Grateful Dead would be honored at the Kennedy Center, I decided to experiment with some AI text or image to video software that had just come out to go along with some music that I had just recorded. So…
First I asked Dall-E (ChatGPT text to image service) to create 18 cartoon images (enough for about 18 10-second videos in a 3 minute song) to accompany the song’s lyrics:
Since it cost a lot to win and even more to lose
You and me bound to spend some time wondering what to choose
Goes to show you don’t ever know
Watch each card you play and play it slow
Wait until your deal come round
Don’t you let that deal go down
I been gambling here abouts for ten good solid years
If I told you all that went down it would burn off both your ears
It goes to show you don’t ever know
Watch each card you play and play it slow
Wait until your deal come round
Don’t you let that deal go down
Since you poured the wine for me and tightend up my shoes I
hate to leave you sittin there composin’ lonesome blues
After first telling me that “Creating cartoon images to accompany the song is a delightful idea!” [thanks for the condescending supportive words, Mr. Chatgpt! ] the AI came quickly came up with 18 ‘scenes’ of its own prompts, from which it created 18 images (not all of which I chose to use.) I didn’t notice until just now (I was focused on the images) that the ‘scenes’ have a somewhat logical storyline of their own, AI having fleshed out or organized an imagined story based on the lyric fragments above. (so AI can imagine…?!) I also just noticed that the AI infused the story with a positive, optimistic ending (not inherent in the lyrics which have a cautionary tone) like: “heading toward a glowing horizon”, “holding a winning hand”, and “sitting on a porch at sunset, smiling.” (so this AI is also optimistic…?! Who knew?)
AI created these ‘Scenes’ or ‘Prompts’ for itself to use to generated images
Scene 1: A dusty Western saloon, with a lone gambler at a poker table, staring at a pile of chips with a look of concern. Cards and coins are scattered on the table.
Scene 2: The gambler is shown walking away from the table, holding his hat in his hands, as the saloon doors swing behind him. A caption reads: “Since it cost a lot to win and even more to lose.”
Scene 3: A crossroads under a night sky, with the gambler standing there, pondering which path to take. A signpost points in different directions with labels like “Win,” “Lose,” and “Unknown.”
Scene 4: A close-up of the gambler shuffling a deck of cards. His eyes are focused, and the caption says: “Watch each card you play and play it slow.”
Scene 5: The gambler is seated at a table with shadowy figures. A glowing card is dealt to him, symbolizing opportunity.
Scene 6: The gambler reaches out, hesitating before picking up the card. The caption reads: “Wait until your deal come round.”
Scene 7: The card glows brighter, as if it’s his destiny. His face shows determination.
Scene 8: The gambler sits in a room, recounting his stories to a wide-eyed audience. The room is filled with the paraphernalia of a gambler’s life: dice, cards, and whiskey bottles. The caption: “I been gambling here abouts for ten good solid years.”
Scene 9: A dramatic flashback scene: cards flying in the air, a table overturned, and the gambler narrowly escaping a chaotic brawl.
Scene 10: The gambler sits alone under a tree at dawn, silhouetted against a pink and orange sky. He looks reflective, holding a single playing card.
Scene 11: The gambler approaches a kind-looking person, who hands him a glass of wine and helps adjust his boots. A small dog sits at their feet.
Scene 12: A cozy, candle-lit room where the gambler and the helper sit together. The gambler looks sad but grateful, the caption reading: “Since you poured the wine for me and tightened up my shoes.”
Scene 13: The gambler walks away from the helper, hat in hand, while the helper watches from the doorway, composing a song on a guitar.
Scene 14: A vivid depiction of the “lonesome blues” being written. The helper sits with a notebook, surrounded by scattered music sheets, a melancholy look on their face.
Scene 15: The gambler is on a new path, heading toward a glowing horizon with a determined stride.
Scene 16: A symbolic image of a hand playing a single card, with the caption: “Don’t you let that deal go down.”
Scene 17: The gambler sits at another poker table, surrounded by shadowy figures again. This time, he smiles confidently, holding a winning hand.
Scene 18: A triumphant, closing image of the gambler and the helper reunited, sitting on a porch at sunset, smiling with a deck of cards between them.
Anyway, first I needed to edit, or rather, asked AI to edit, a few iterations of some images, especially on the “burn off both your ears” image. Then I chose the best images to use for image to video production.
Initially I tried Sora, ChatGPT’s text/image to video service. It was too cumbersome and had some limitations. Then I tried RunwayML, which had also just come out and incidentally, recently signed a deal with a major Hollywood studio). On RunwayML I spent $15 of credits for the whole project, which involved importing the previously created images, having the opportunity to input another text prompt, and deciding to create either 5 or 10 second videos (more dólares for more length of course).
Once the clips were created, the desired ones were imported into Final Cut Pro and timed to match the previously recorded multitrack version of Deal. I also shared the audio recording with my friend, guitarist Lee Offenhauer. After all, this song having been written by Jerry absolutely needed an authentic guitar solo, and Lee came up with a very tasteful one which I then mixed in with the other tracks.
Enjoy,
Dr. Weiss
Deadicated to Mark C and Sean W
Philosophical Questions: So, will human creativity in the future simply be creating the most interesting, detailed and creative prompts – and then using AI to create further improvements of its own outputs? Scientist and futurist Ray Kurzweil coined the term the ‘singularity’ (an era of almost instantaneous change especially when AI starts to code its own improvements, among other things) and has written ‘The Singularity is Near’ in 2005 and ‘The Singularity is Nearer’ in 2024. Are we actually starting to live through it already? |
Comments are closed.