Issue #, if available: #75

Description of changes: This is a draft pull request to add LBFGS optimizer for optimization of deterministic loss functions. I found this works much better than the current default optimizer for vanilla GPs.

If you don't object to the approach I use here I will take the time and add tests and tidy the code up.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The module algorithms dictionary is expected to be a list containing tuples but after cloning a module this is not the case. This PR fixes that.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This stops the debugger accidentally creating new variables on access.

Description of changes: When using an interactive debugger (e.g. PyCharm), inspecting any Variable object invokes the property T which has the side effect of creating new variables that are not part of the graph. By changing this to a function .transpose() the functionality is maintained (at the slight expense of ease of use) whilst preventing this from happening.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This fixes #183 (but maybe you want to copy kernels in a different way as discussed yesterday). It also fixes a few other issues with cloning this specific GP module and adds tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

This PR adds an implementation of DGPs.

I have added a marginalise_conditional_distribution method to the conditional GP class which is used both by the deep GP and the SVI GP.

I chose not to explicitly represent the conditional p(f|u) as I couldn't work out how to represent it while making the marginalization as efficient.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Description of changes: Shift MXNet based constants from being stored in the InferenceParameters._constants dictionary (which still exists and keeps scalar/native constants) to keeping MXNet constants in the ParameterDict where other Parameters are kept.

This removes the need for the separate MXNet constants file at serialization file as well!

(Feel free to ignore the documentation change here, it's a remnant of the larger docs change I made in another PR and I'll try to make sure the other one is the one that gets kept.)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

These should be implemented as stateless functions.

Create a class that takes in variables and returns a function evaluation.

Basic - Add, subtract, multiplication, division of variables.

elementwise - square, exponentiation, log

Aggregation - sum, mean, prod

Matrix ops - dot product, diag

Matrix manipulation - reshape, transpose

It contains the following changes:

• Folder structure changes and renaming according to our discussion.
• Extend the Mark's multivariate normal distribution implementation to allow general broadcasting.

@marpulli I changed the behavior of your log_pdf and kl_divergence implementation to return an array of the size of mean without the last dimension. This is consistent with our other distributions.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

The monthly cron build of Travis-CI on master branch fails, because it tries to deploy again. Here is the link to a build: https://travis-ci.org/amzn/MXFusion/jobs/453598240.

Issue #, if available:

Description of changes:

Uniform and Laplace distributions + tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue #, if available: #21

Description of changes: Added Bernoulli distribution + tests

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Hi,

I'm a member of Deep Numpy (a new MXNet frontend API with numpy like interface: https://numpy.mxnet.io/index.html) dev team and I'm currently working on the random module for Deep Numpy, which, as its name, has behavior exactly like Numpy, including broadcastable parameters and output shape, for example: https://github.com/apache/incubator-mxnet/pull/15858

Also, sampling backend for rejection sampling is entirely rewritten to remove while loop from GPU Kernel. The new version, according to my profiling, performs ten times faster than nd.random on GPU at the cost of tiny increase in memory usage. (not merged yet) https://github.com/apache/incubator-mxnet/issues/15928

We could have some further discussion if you have interests in switching to deep numpy's random sampling backend, or, migrating MXFusion to Deep Numpy API :-)

Is your feature request related to a problem? Please describe. The SVGP and DGP modules should be able to work with alternative likelihoods. They are both currently implemented for Gaussian likelihoods only. The ability to switch out likelihoods would make things like #126 much simpler with less code duplication.

Describe the solution you'd like Currently:

m = Model()
m.X = Variable()
m.noise_var = Variable(transformation=PositiveTransformation())
m.Y = SVGP.define_variable(m.X, kern, m.noise_var...)

I'd like something more like

m = Model()
m.X = Variable()
m.noise_var = Variable(transformation=PositiveTransformation())
m.likelihood = NormalGPLikelihood(variance)
# This could also be: 
# m.likelihood = BernoulliGPLikelihood()
m.Y = SVGP.define_variable(m.X, kern, m.likelihood...)

These likelihoods would need to be able to compute: where p(f) is Gaussian.

class BernoulliGPLikelihood(GPLikelihood):
    # This class would also contain the transformation of the 
    # latent function to the interval [0, 1] I think
   def expectation(self, mean, variance, data): 
       # This method would implement the quadrature rule to do this here

class NormalGPLikelihood(GPLikelihood):
   def expectation(self, mean, variance, data): # think of a better name!
       # This method could use the analytically equation

Describe alternatives you've considered Creating a new class might not be necessary, we might be able to reuse the current distribution objects directly.

I would be happy to implement this, if we come up with a design which people are happy with first.

Describe the bug

When passing Variables of type PARAMETER as 'constants' argument to the constructor of Inference (or child classes), this line throws during the execution of Inference's run method since the Variables that were passed as 'constants' are in self._var_trans but not in the 'kw' input argument.

Expected behavior

Execution does not throw and Variables passed as 'constants' argument to the Inference constructor, even if they are of type PARAMETER, are treated as constants and not optimized in training.

Desktop:

• OS: Ubuntu 18.04.2
• Python version: 3.6
• MXNet version: 1.4.1
• MXFusion version: 0.3.1
• MXNet context: CPU

Describe the bug

If a kernel Variable is of CONSTANT type, fetch_parameters throws here since CONSTANT Variables do not make it to params input argument.

Expected behavior

Execution does not throw, and kernel Variables that are of type CONSTANT are kept constant and not optimized in training.

Desktop:

• OS: Ubuntu 18.04.2
• Python version: 3.6
• MXNet version: 1.4.1
• MXFusion version: 0.3.1
• MXNet context: CPU

Describe the bug

If batch_size for MinibatchInferenceLoop is larger than the dataset size, this line throws due to division by zero.

Expected behavior

Possible mitigations are:

• Option No. 1: Default batch size to min("requested batch size", "dataset size") and issue a warning if "requested batch size" > "dataset size".
• Option No. 2: Throw with a message stating why it's throwing and how to fix the issue.

Desktop:

• OS: Ubuntu 18.04.2
• Python version: 3.6
• MXNet version: 1.4.1
• MXFusion version: 0.3.1
• MXNet context: CPU