Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Last update: Mar 15, 2022

Overview

FeatureHasher

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Note, this requires Jina>=2.2.4.

Example

Here I use FeatureHasher to hash each sentence of Pride and Prejudice into a 128-dim vector, and then use .match to find top-K similar sentences.

from jina import Document, DocumentArray, Flow

# load 
   
d = Document(uri='https://www.gutenberg.org/files/1342/1342-0.txt').convert_uri_to_text()

# cut into non-empty sentences store in a DA
da = DocumentArray(Document(text=s.strip()) for s in d.text.split('\n') if s.strip())

# use FeatureHasher in a Flow
f = Flow().add(uses='jinahub://FeatureHasher')

embed_da = DocumentArray()
with f:
    f.post('/', da, on_done=lambda req: embed_da.extend(req.docs), show_progress=True)

print('self-matching...')
embed_da.match(embed_da, exclude_self=True, limit=5, normalization=(1, 0))
print('total sentences: ', len(embed_da))
for d in embed_da:
    print(d.text)
    for m in d.matches:
        print(m.scores['cosine'], m.text)
    input()

           [email protected][I]:🎉 Flow is ready to use!
	🔗 Protocol: 		GRPC
	🏠 Local access:	0.0.0.0:52628
	🔒 Private network:	192.168.178.31:52628
	🌐 Public address:	217.70.138.123:52628
⠹       DONE ━╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:01 100% ETA: 0 seconds 40 steps done in 1 second
total sentences:  12153
The Project Gutenberg eBook of Pride and Prejudice, by Jane Austen

   
     *** END OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

    
      *** START OF THE PROJECT GUTENBERG EBOOK PRIDE AND PREJUDICE ***

     
       production, promotion and distribution of Project Gutenberg-tm

      
        Pride and Prejudice

       
         By Jane Austen This eBook is for the use of anyone anywhere in the United States and 
        
          This eBook is for the use of anyone anywhere in the United States and 
         
           by the awkwardness of the application, and at length wholly 
          
            Elizabeth passed the chief of the night in her sister’s room, and 
           
             the happiest memories in the world. Nothing of the past was 
            
              charities and charitable donations in all 50 states of the United

In practice, you can implement matching and storing via an indexer inside Flow. This example is only for demo purpose so any non-feature hashing related ops are implemented outside the Flow to avoid distraction.

Convert a collection of features to a fixed-dimensional matrix using the hashing trick.

Related tags

Overview

FeatureHasher

Example

Owner

Jina AI

DomainMonitor is a web project that has a RESTful API to get a domain's subdomains and whois data.

SSH Tool For OSINT and then Cracking.

Discord Region Swapping Exploit (VC Overload)

A simple password generator using Python Tkinter.

DoSer.py - Simple DoSer in Python

Arbitrium is a cross-platform, fully undetectable remote access trojan, to control Android, Windows and Linux and doesn't require any firewall exceptions or port forwarding rules

IDA Pro Python plugin to analyze and annotate Linux kernel alternatives

Template for new OSINT command-line tools

Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples

Gefilte Fish GMail filter creator

Rouge Spammers with a mission to disrupt the peace of the valley ? Fear not we will STOMP the Spammers

宝塔面板Windows版提权方法

Local server for IDA Lumina feature

Python HDFS client

Fast subdomain scanner, Takes arguments from a Json file ("args.json") and outputs the subdomains.

对安卓APP注入MSF PAYLOAD，并且对手机管家进行BYPASS。

Simple Python 3 script to detect the "Log4j" Java library vulnerability (CVE-2021-44228) for a list of URL with multithreading

Unauthenticated Sqlinjection that leads to dump data base but this one impersonated Admin and drops a interactive shell

This repository uses a mixture of numbers, alphabets, and other symbols found on the computer keyboard

A quick script to spot the usage of Unicode Bidi (bidirectional) characters that could lead to an Invisible Backdoor