NASH 2021 project... this may or may not end up working 🤷‍♂️

Overview

wavespace synthesiser

this is my NASH 2021 project, which may or may not end up working 🤷‍♂️

what is going on?

imagine you have a big folder of audio samples. wouldn't it be cool if you could seamlessly morph between them according to some axes of control you define?

that's the idea here: to have a synthesiser which encodes the timbral characteristics of all your audio. it works by learning continuous representations of the harmonic and aperiodic components of the audio signals using conditional neural fields.

what on earth are neural fields?

first, fields. in the sense the term is used in physics, a field is some quantity (a measurement of some physical quantity, for example) that has values across some system of spatiotemporal co-ordinates. a neural field is simply a neural network that parameterises a field. you can think of it like a network that takes in co-ordinates and outputs the value you're interested in. for example, you might pass in spatial (x, y) co-ordinates and get out the pixel values (r, g, b) of an image. or, you might pass in temporal co-ordinates (t) and get out the amplitude value (a) of an audio signal.

there are a few ways to achieve this with a neural network. the most successful recent methods use periodic nonlinearities or basis functions, such as SIREN, multiplicative filter networks, and MLPs with Fourier features. here, we use SIRENs, but there is no reason other methods wouldn't work.

here we stretch the definition of a neural field slightly. our co-ordinate systems are not always spatiotemporal, and our measurements are not always physical quantities. we use two neural fields for the model in this repo:

  1. one which learns a continuous space of perfectly looping waveforms (kind of like a bank of wavetables, but without the table and without the bank, hence wavespace) as a function of time and phase
  2. one which learns a continuous space of zero-phase FIR filter magnitude responses as a function of time and frequency bin

in theory, these continuous representations should allow us to do fun things like sampling at multiple resolutions.

ok, sure. but what is a conditional neural field?

well, typically you fit a neural field to a single signal. there is some work looking at learning priors over the space of weights of neural fields to allow them to generalise, but it's quite hard to do (i will be releasing a paper soon on some reasons for this). in our scenario, though, we want to be able to represent all the sounds in our dataset continuously, which a single neural field lacks the capacity to do.

so, we turn to our trusty friend FiLM conditioning for help. we use a separate network to generate the FiLM parameters which are inserted between the layers of our neural field, subtly shifting the activations to achieve vastly different results, whilst hopefully allowing the shared weights to learn a convenient general representation. i'm not the first to consider this approach for audio synthesis: a recent paper on FiLM conditioned SIRENs for audio synthesis had promising results. however, synthesising audio samples directly is hard. imposing useful priors like the harmonicity of wavetable synthesis, along with a harmonic-plus-noise signal decomposition, allows us to reduce the complexity of the network's task.

how does it sound?

more soon!

and what is the model architecture?

also more soon!

Owner
Ben Hayes
AI & Music PhD researcher @ Centre for Digital Music, QMUL
Ben Hayes
A test repository to build a python package and publish the package to Artifact Registry using GCB

A test repository to build a python package and publish the package to Artifact Registry using GCB. Then have the package be a dependency in a GCF function.

1 Feb 09, 2022
A replacement of qsreplace, accepts URLs as standard input, replaces all query string values with user-supplied values and stdout.

Bhedak A replacement of qsreplace, accepts URLs as standard input, replaces all query string values with user-supplied values and stdout. Works on eve

Eshan Singh 84 Dec 31, 2022
A basic interpreted programming language written in python

shin A basic interpreted programming language written in python. extension You can use our own extension ".shin". Example: main.shin How to start Clon

12 Nov 04, 2022
Job Guy Backend

جاب‌گای چیست؟ اونجا وضعیت چطوریه؟ یه سوال به همین کلیت و ابهام معمولا وقتی برای یه شرکت رزومه می‌فرستیم این سوال کلی و بزرگ برای همه پیش میاد.اونجا وض

Jobguy.work 217 Dec 25, 2022
NCAR/UCAR virtual Python Tutorial Seminar Series lesson on MetPy.

The Project Pythia Python Tutorial Seminar Series continues with a lesson on MetPy on Wednesday, 2 February 2022 at 1 PM Mountain Standard Time.

Project Pythia Tutorials 6 Oct 09, 2022
Aerospace utilities: flight conditions package, standard atmosphere model, and more.

Aerospace Utilities About Module that contains commonly-used aerospace utilities for problem solving. Flight Condition: input altitude to compute comm

1 Jan 03, 2022
A script for creating battle animations in FEGBA format.

AA2 Made by Huichelaar. I heavily referenced FEBuilderGBA. I also referenced circleseverywhere's Animation Assembler. This is also where I took lzss.p

2 May 31, 2022
Christmas tree on the desktop.

new-year-tree Christmas tree on the desktop. [Ссылка на статью habr]

Daniil Gorbenko 10 Dec 29, 2022
An easy FASTA object handler, reader, writer and translator for small to medium size projects without dependencies.

miniFASTA An easy FASTA object handler, reader, writer and translator for small to medium size projects without dependencies. Installation Using pip /

Jules Kreuer 3 Jun 30, 2022
This repository can help you made a PocketMine-MP Server with Termux apps!

Hello This GitHub repository can made you a Server PocketMine-MP On development! How to Install Open Termux Type "pkg install git && python" If python

1 Mar 04, 2022
Python script to preprocess images of all Pokémon to finetune ruDALL-E

ai-generated-pokemon-rudalle Python script to preprocess images of all Pokémon (the "official artwork" of each Pokémon via PokéAPI) into a format such

Max Woolf 132 Dec 11, 2022
Necst-lib - Pure Python tools for NECST

necst-lib Pure Python tools for NECST. Features This library provides: something

NANTEN2 Group 5 Dec 15, 2022
UdemyPy is a bot that hourly looks for Udemy free courses and post them in my Telegram Channel: Free Courses.

UdemyPy UdemyPy is a bot that hourly looks for Udemy free courses and post them in my Telegram Channel: Free Courses. How does it work? For publishing

88 Dec 25, 2022
MindF**k it's a programming language as BrainFuck, but with some cool features.

MindF**k Description MindF**k it's a programming language as BrainFuck, but with some cool features. Symbol What does symbol mean Next slot Previo

tixcode 0 Jun 15, 2022
Gmvault: Backup and restore your gmail account

Gmvault: Backup and restore your gmail account Gmvault is a tool for backing up your gmail account and never lose email correspondence. Gmvault is ope

Guillaume Aubert 3.5k Jan 01, 2023
Python dictionaries with advanced dot notation access

from box import Box movie_box = Box({ "Robin Hood: Men in Tights": { "imdb stars": 6.7, "length": 104 } }) movie_box.Robin_Hood_Men_in_Tights.imdb_s

Chris Griffith 2.1k Dec 28, 2022
A proof-of-concept package manager for Cairo contracts/libraries

glyph A proof-of-concept package manager for Cairo contracts/libraries. Distribution through pypi. Installation through existing package managers -- p

Sam Barnes 11 Jun 06, 2022
Euler 021 Py - Euler Problem 021 solved in Python

Euler_021_Py Euler Problem 021 solved in Python Let d(n) be defined as the sum o

Ariel Tynan 1 Jan 24, 2022
This project recreates the R-based RCy3 Cytoscape Automation library as a Python package.

Python library for calling Cytoscape Automation via CyREST

Cytoscape Consortium 40 Dec 22, 2022
A performant state estimator for power system

A state estimator for power system. Turbocharged with sparse matrix support, JIT, SIMD and improved ordering.

9 Dec 12, 2022