Lucid Sonic Dreams syncs GAN-generated visuals to music.

Overview

Lucid Sonic Dreams

Lucid Sonic Dreams syncs GAN-generated visuals to music. By default, it uses NVLabs StyleGAN2, with pre-trained models lifted from Justin Pinkney's consolidated repository. Custom weights and other GAN architectures can be used as well.

Sample output can be found on YouTube and Instagram.

Installation

This implementation has been teston on Python 3.6 and 3.7. As per NVLabs' TensorFlow implementation of StyleGAN2, TensorFlow 1.15 is required. TensorFlow 2.x is not supported.

To install, simply run:

pip install lucidsonicdreams

Usage

You may refer to the Lucid Sonic Dreams Tutorial Notebook for full parameter descriptions and sample code templates. A basic visualization snippet is also found below.

Basic Visualization

from lucidsonicdreams import LucidSonicDream


L = LucidSonicDream(song = 'song.mp3',
                    style = 'abstract photos')

L.hallucinate(file_name = 'song.mp4') 
Comments
  • where to place .pkl model files?

    where to place .pkl model files?

    Hi,

    Thanks for the fantastic repo,

    I really want to drop a custom .pkl model file into LucidSonicDreams and it isn't obvious to me where I should put it? I'm working in Colab for the time being.

    Thanks,

    Mark

    opened by markhanslip 1
  • Installing in Ubuntu 20.04 ?

    Installing in Ubuntu 20.04 ?

    Hi I tried now for hours to install Lucid Sonic Dreams in Ubuntu 20.04. How to install it correctly so that it works ? I tried it with anaconda but no luck....a little desperate now ! Update: Installed everything without errors. Used the setup.py to install dependencies. But now i am stuck. Where and how to execute this:

    from lucidsonicdreams import show_styles

    Show valid default style names. show_styles()

    or this ?

    "from lucidsonicdreams import LucidSonicDream

    L = LucidSonicDream(song = 'song.mp3', style = 'abstract photos')

    L.hallucinate(file_name = 'song.mp4') " ??

    Can someone enlighten me please ?

    opened by Colliwomple 0
  • Fix for broken Deps possibly?  Please advise!  LSD colab is BROKEN!  Thanks!

    Fix for broken Deps possibly? Please advise! LSD colab is BROKEN! Thanks!

    See - https://github.com/mikaelalafriz/lucid-sonic-dreams/compare/main...pollinations:lucid-sonic-dreams:main suggestion for pollinations to mod to self refer so their fixes they made can be used, otherwise its referring to the same broken changes that you have that are breaking the colabs for LSD.

    [fuse bias errors mostly to do with incompatibilities in breaking changes to several depenancies and potential v2 v3 python issues with v1/v2 tensorflow.]

    ITs fixable but we need to specify the old working dependencies from what i can see, not the new breaking ones. All this began after the default code attempted to integrate ADA from what i could see? Correct me if i am wrong thanks!

    opened by cleancoindev 1
  • Real time support

    Real time support

    Hi,

    First, thank you for your great work - it's incredible!

    I was wondering if, in your opinion, it would be possible to extend your work to generate the visuals in real-time. This would mean using streaming of audio data (or, possibly, MIDI) rather than pre-rendered files. I guess the frame rate can be a little low at 1024, but it would be still great to have this option for someone who has a lot of GPUs. Do you think it would be anyhow realistic?

    Keep up the amazing work!

    opened by lowlypalace 0
  • ModuleNotFoundError: No module named 'lucidsonicdreams'

    ModuleNotFoundError: No module named 'lucidsonicdreams'

    Im trying to run a test and this is the way i have the python file typed. Any help would be appreciated

    (command i input)= python proud.py (to run the python below)

    from lucidsonicdreams import LucidSonicDream

    L = LucidSonicDream(song = 'proud.mp3', style = 'abstract photos')

    L.hallucinate(file_name = 'proud.mp4', resolution = 360, start = 30, duration = 45)

    files.download("proud of you.mp4")

    Error im getting

    Traceback (most recent call last): File "proud.py" line 1 in from lucidsonicdreams import LucidSonicDream ModuleNotFoundError: No module named 'lucidsonicdreams

    Screenshot (4) '

    opened by Texagon 3
  • index out of bounds

    index out of bounds

    Hi! I am trying out the script in order to sync some images I have generated using VQGAN+CLIP to my audio. Here's the code:

    def load_imgs(noise_batch, class_batch):
        # just loads N images randomly
        return images
    
    L = LucidSonicDream('audio_5.mp3',
                        style = load_imgs, 
                        input_shape = 592,
                        num_possible_classes = 1000)
    
    L.hallucinate('video_sync.mp4',
                  output_audio = 'audio_sync.mp3',
                  speed_fpm = 3,
                  classes = [13, 14, 22, 24, 301, 84, 99, 100, 134, 143, 393, 394],
                  class_shuffle_seconds = 10, 
                  class_shuffle_strength = 0.1,
                  class_complexity = 0.5,
                  class_smooth_seconds = 4,
                  motion_react = 0.35,
                  flash_strength = 1)
                  #contrast_strength = 0.5)
    

    The error appears just at the end of the process:

    IndexError                                Traceback (most recent call last)
    <ipython-input-15-aeedb41e1387> in <module>()
         15               class_smooth_seconds = 4,
         16               motion_react = 0.35,
    ---> 17               flash_strength = 1)
         18               #contrast_strength = 0.5)
    
    2 frames
    /usr/local/lib/python3.7/dist-packages/lucidsonicdreams/main.py in apply_effect(self, array, index)
        742     '''Apply effect to image (array)'''
        743 
    --> 744     amplitude = self.spec[index]
        745     return self.func(array=array, strength = self.strength, amplitude=amplitude)
    
    IndexError: index 207 is out of bounds for axis 0 with size 207
    

    Any idea on how to avoid it? Thanks in advance!

    opened by shoegazerstella 0
Releases(v_04)
This is a collection of simple PyTorch implementations of neural networks and related algorithms. These implementations are documented with explanations,

labml.ai Deep Learning Paper Implementations This is a collection of simple PyTorch implementations of neural networks and related algorithms. These i

labml.ai 16.4k Jan 09, 2023
A solution to the 2D Ising model of ferromagnetism, implemented using the Metropolis algorithm

Solving the Ising model on a 2D lattice using the Metropolis Algorithm Introduction The Ising model is a simplified model of ferromagnetism, the pheno

Rohit Prabhu 5 Nov 13, 2022
Improving Transferability of Representations via Augmentation-Aware Self-Supervision

Improving Transferability of Representations via Augmentation-Aware Self-Supervision Accepted to NeurIPS 2021 TL;DR: Learning augmentation-aware infor

hankook 38 Sep 16, 2022
Classifies galaxy morphology with Bayesian CNN

Zoobot Zoobot classifies galaxy morphology with deep learning. This code will let you: Reproduce and improve the Galaxy Zoo DECaLS automated classific

Mike Walmsley 39 Dec 20, 2022
IJON is an annotation mechanism that analysts can use to guide fuzzers such as AFL.

IJON SPACE EXPLORER IJON is an annotation mechanism that analysts can use to guide fuzzers such as AFL. Using only a small (usually one line) annotati

Chair for Sys­tems Se­cu­ri­ty 146 Dec 16, 2022
tensorflow code for inverse face rendering

InverseFaceRender This is tensorflow code for our project: Learning Inverse Rendering of Faces from Real-world Videos. (https://arxiv.org/abs/2003.120

Yuda Qiu 18 Nov 16, 2022
Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models.

WECHSEL Code for WECHSEL: Effective initialization of subword embeddings for cross-lingual transfer of monolingual language models. arXiv: https://arx

Institute of Computational Perception 45 Dec 29, 2022
Share a benchmark that can easily apply reinforcement learning in Job-shop-scheduling

Gymjsp Gymjsp is an open source Python library, which uses the OpenAI Gym interface for easily instantiating and interacting with RL environments, and

134 Dec 08, 2022
A curated list of resources for Image and Video Deblurring

A curated list of resources for Image and Video Deblurring

Subeesh Vasu 1.7k Jan 01, 2023
The Illinois repository for Climatehack (https://climatehack.ai/). We won 1st place!

Climatehack This is the repository for Illinois's Climatehack Team. We earned first place on the leaderboard with a final score of 0.87992. An overvie

Jatin Mathur 20 Jun 09, 2022
Dist2Dec: A Simplicial Neural Network for Homology Localization

Dist2Dec: A Simplicial Neural Network for Homology Localization

Alexandros Keros 6 Jun 12, 2022
Cross-platform CLI tool to generate your Github profile's stats and summary.

ghs Cross-platform CLI tool to generate your Github profile's stats and summary. Preview Hop on to examples for other usecases. Jump to: Installation

HackerRank 134 Dec 20, 2022
Randstad Artificial Intelligence Challenge (powered by VGEN). Soluzione proposta da Stefano Fiorucci (anakin87) - primo classificato

Randstad Artificial Intelligence Challenge (powered by VGEN) Soluzione proposta da Stefano Fiorucci (anakin87) - primo classificato Struttura director

Stefano Fiorucci 1 Nov 13, 2021
A way to store images in YAML.

YAMLImg A way to store images in YAML. I made this after seeing Roadcrosser's JSON-G because it was too inspiring to ignore this opportunity. Installa

5 Mar 14, 2022
Text to Image Generation with Semantic-Spatial Aware GAN

text2image This repository includes the implementation for Text to Image Generation with Semantic-Spatial Aware GAN This repo is not completely. Netwo

CVDDL 124 Dec 30, 2022
Multi Task RL Baselines

MTRL Multi Task RL Algorithms Contents Introduction Setup Usage Documentation Contributing to MTRL Community Acknowledgements Introduction M

Facebook Research 171 Jan 09, 2023
[ICCV 2021 Oral] Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Just Ask: Learning to Answer Questions from Millions of Narrated Videos Webpage • Demo • Paper This repository provides the code for our paper, includ

Antoine Yang 87 Jan 05, 2023
Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Zhensu Sun 1 Oct 26, 2021
Library of deep learning models and datasets designed to make deep learning more accessible and accelerate ML research.

Tensor2Tensor Tensor2Tensor, or T2T for short, is a library of deep learning models and datasets designed to make deep learning more accessible and ac

12.9k Jan 09, 2023
This is a vision-based 3d model manipulation and control UI

Manipulation of 3D Models Using Hand Gesture This program allows user to manipulation 3D models (.obj format) with their hands. The project support bo

Cortic Technology Corp. 43 Oct 23, 2022