Example about non-linear starts stabilizing IMO, so I can start taking feedback.

In short, the example shows a classification problem for the traffic_violations dataset.

5 types of models are implemented:

• SimilarityEncoder + SVC
• SimilarityEncoder + RBFSampler + SVC
• SimilarityEncoder + RBFSampler + SGD
• SimilarityEncoder fitted on a subset of the data + RBFSampler + SGD
• SimilarityEncoder fitted on a subset of the data + RBFSampler + online SGD

Training set sizes still dummy to make CI pass fast. For now, the scripts takes around 15 sec to run. RBFSampler can easily be switched in favor of Nystroem.

I wanted to improve the fuzzy_join example to first use fuzzy_join as it would be used by people using pandas outside of a predictive analysis, and then do the predictive analysis: hence separating X and y later.

I hit a problem that dtypes are not maintained by the merge. I created a simple failing test to illustrate the problem.

Follows #147

This is a refactor of the dataset fetching system. Historically, datasets were fetched from various websites.

Our aim with this update is to only use OpenML.org's API, through Scikit-learn's fetch_openml() function. This allows us to have a much more reliable and unified interface, and avoids losing access to datasets due to deletion, renaming, etc.

For instance, with the current system, 4 on the 7 datasets are unavailable (403 Access Denied, website down, etc.).

The user-interface stays the same with the functions fetch_*() (e.g fetch_open_payments()), still returning a similar dictionary. The major difference is that this dictionary returns, among other information, a path, where a CSV file is located, and must be loaded (using for instance pandas' read_csv() function).

TL;DR of the previous thread: the way fetch_openml() is used here makes pandas a requirement.

Changes:

• Added gamma_poisson_factorization.py which implements online Gamma-Poisson factorization for encoding string variables.
• Added the corresponding tests intest_gamma_poisson_factorization.
• Modified examples 02 and 03 to include this method.
• Updated CHANGES.rst and index.rst to describe this new method.

Encoding a missing value as a vector of zeros is a reasonable thing. Our theoretical study (https://arxiv.org/abs/1902.06931) shows that the most important thing is to encode them in a special value that can be later picked up by the supervised step.

Our encoders should have an option that controls whether missing values are encoded as zeros or an error is raised (following scikit-learn encoders).

This PR aims at improving the overall quality of the code and doc.

It has several purposes:

• Correct typos
• Reword unclear sentences
• Minor updates to the doc
• Some minor structural improvements, such as moving some functions to suiting modules
• Rename some variables for better readability
• Use modern language features for better readability and performance
• Simplify the code, while leaving functionalities intact (no bug-fixes)
• Make extensive use of type hinting, which serves two purposes:
  • Make the code easier to work with, especially when working with IDEs that support type hinting
  • Make the functions more efficient and less error-prone when using tools that enforce types, such as MyPy

In general, these are rather small modifications for which making unique PRs would be kind of overkill.

Compute the min hash transform method in parallel, as suggested by @alexis-cvetkov.

We no longer use the self.hash_dict attribute, so the fit method does nothing now.

Hy all, I am trying to apply minhash_encoder to a somewhat large dataset of strings (~200k distinct). I was testing my code with 10 strings, and it was running fine. But when I tested using all dataset, most of the strings were represented as all '-1' vectors. I took a look at the source code and find this line inside 'minhash_encoder.py', that maybe is causing the problem: self.hash_dict = LRUDict(capacity=2**10) Not sure why this is used, but I checked with 1025 strings, and only the first one returns -1. This encoder should work with a lot more variables, right?

Code to replicate:

from dirty_cat import MinHashEncoder
import random
import string

def get_random_string(length):
    letters = string.ascii_lowercase
    result_str = ''.join(random.choice(letters) for i in range(length))
    return result_str

# 1024 categories -> all ok
raw_data = [get_random_string(10) for x in range(1024)]
hash_encoder = MinHashEncoder(n_components=10)
transformed_values = hash_encoder.fit_transform(raw_data)
print(transformed_values)

# 1025 categories -> first represented as -1's
raw_data = [get_random_string(10) for x in range(1025)]
hash_encoder = MinHashEncoder(n_components=10)
transformed_values = hash_encoder.fit_transform(raw_data)
print(transformed_values)

Hello!

I was trying to reproduce "Investigating dirty categories" (https://dirty-cat.github.io/stable/auto_examples/01_investigating_dirty_categories.html#sphx-glr-auto-examples-01-investigating-dirty-categories-py) and got this error: AttributeError: 'tuple' object has no attribute 'shape'.

Log says it is in line 241, in fit n_samples, n_features = X.shape

Am I doing something wrong or is it a issue?

I'm on python 3.7.

Thanks

Accelerate the computation in SimilarityEncoder.transform by:

• Parallelizing the similarity computations using joblib
• Computing the count vectors of the vocabulary at fitting time and not at transform time.

Actual behavior

The Super Vectorizer transform and fit_transform methods have the following rule: "If any result is a sparse matrix, everything will be converted to sparse matrices." This is the scipy.sparse.csr.csr_matrix type.

However, this type is not commonly accepted for further analysis. For instance, when applying a cross_val_score() we need to first make the result an array to be able to apply the method. This makes also the direct introduction of pipelines in cross_val_score() impossible, as an error will appear.

Expected behavior

Sparse matrices happen when the encoded variable has a lot of categories. Maybe introduce a sparse=True parameter, just like for the sklearn OHE, that will return sparse matrix if set True and array if False.

Easy code to reproduce bug

import pandas as pd
import numpy as np

from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
from sklearn.experimental import enable_hist_gradient_boosting
# now you can import the HGBR from ensemble
from sklearn.ensemble import HistGradientBoostingRegressor
from dirty_cat import SuperVectorizer

np.random.seed(444) 
col1 = np.random.choice(  
     a=[0, 1, 2, 3],  
     size=50,  
     p=[0.4, 0.3, 0.2, 0.1])

col2 = np.random.choice(  
     a=['a', 'b', 'c'],  
     size=50,  
     p=[0.4, 0.4, 0.2])

results = np.random.uniform( 
     size=50)

df = pd.DataFrame(np.array([col1, col2, results])).transpose()

X = df.drop(columns=[2])
y = df[2]

sup_vec = SuperVectorizer()

pipeline = make_pipeline(
    SuperVectorizer(auto_cast=True, sparse_threshold=0.3),
    HistGradientBoostingRegressor()
)

cross_val_score(pipeline, X, y)

Following #446 (which seems to show that using HashingVectorizer is almost always faster than using CountVectorizer, without any apparent accuracy tradeoff) and discussion, this PR adds a vectorizer parameter to the fuzzy_join function, which defaults to hashing, i.e using HashingVectorizer.

Replaces #420.

I think someone should check my benchmark in #446 before we consider merging this PR.

• Add the possibility for the benchmark function to return a dictionnary, which is added to the results by the monitor decorator • Benchmark different encoders for fuzzy_join (issue #418, related to #420) . It seems that using the HashingVectorizer instead of CountVectorizer is always better (no cost for f1 score, and almost always faster, see plot). If the user want to tradeoff performance for speed, it seems better to play with the ngram_range than changing the encoder. Therefore I recommend to just use the HashingVectorizer for fuzzy_join, instead of using it only for big datasets like in #420.

Looking forward to hearing what other people think!

SimilarityEncoder does not raise parameter Value Error at initialisation. So the user realise there is a problem only after trying to fit the encoder.

dirty_cat version:

Expected behavior:

SimilarityEncoder(handle_unknown='blabla')
___________________________________________________________________________________
ValueError: Got handle_unknown='blabla', but expected any of {'error', 'ignore'}. 

Observed behavior:

SimilarityEncoder(handle_unknown='blabla')
_______________________________________________
# No errors

Sorry for the annoying to review PR! It's a bunch of changes I had in a leftover branch. Thought it would still be useful to push. Some changes are redundant with #426.