The evaluator covering all of the metrics required by tasks within the DUE Benchmark.

Last update: Jan 21, 2022

Related tags

Overview

DUE Evaluator

The repository contains the evaluator covering all of the metrics required by tasks within the DUE Benchmark, i.e., set-based F1 (for KIE), ANLS (used in document VQA), accuracy (including variant used in WTQ), as well as group-based ANLS we proposed for KIE problems with structured output.

Usage

The deval command will be available after the package installation. Every time, it is required to provide input and output files (both in the DU-Schema format) using -o and -r parameters.

Other settings are task-specific and limited to metric (-m) and optional case-insensitiveness (-i). Recommended values of these are:

Dataset	Metric	Case insensitive
DocVQA, InfographicsVQA	ANLS	Yes
Kleister Charity, DeepForm	F1	Yes
PapersWithCode	GROUP-ANLS	Yes
WikiTableQuestions	WTQ	No (handled by metric itself)
TabFact	F1 (obtained value will be equal to Accuracy)	No

Owner

DUE Benchmark

The benchmark consisting of both available and reformulated datasets to measure the end-to-end capabilities of systems in real-world scenarios.

GitHub Repository

Hypothesis is a powerful, flexible, and easy to use library for property-based testing.

Hypothesis Hypothesis is a family of testing libraries which let you write tests parametrized by a source of examples. A Hypothesis implementation the

6.4k Jan 05, 2023

Argument matchers for unittest.mock

callee Argument matchers for unittest.mock More robust tests Python's mocking library (or its backport for Python 3.3) is simple, reliable, and easy

77 Nov 03, 2022

Rerun pytest when your code changes

A simple watcher for pytest Overview pytest-watcher is a tool to automatically rerun pytest when your code changes. It looks for the following events:

74 Dec 29, 2022

Data App Performance Tests

Data App Performance Tests My hypothesis is that The different architectures of

6 Dec 14, 2022

API Test Automation with Requests and Pytest

api-testing-requests-pytest Install Make sure you have Python 3 installed on your machine. Then: 1.Install pipenv sudo apt-get install pipenv 2.Go to

2 Nov 21, 2021

自动化爬取并自动测试所有swagger-ui.html显示的接口

swagger-hack 在测试中偶尔会碰到swagger泄露常见的泄露如图：有的泄露接口特别多，每一个都手动去试根本试不过来于是用python写了个脚本自动爬取所有接口，配置好传参发包访问原理是首先抓取http://url/swagger-resources 获取到有哪些标准及对应的文档地

534 Dec 29, 2022

pytest plugin for manipulating test data directories and files

pytest-datadir pytest plugin for manipulating test data directories and files. Usage pytest-datadir will look up for a directory with the name of your

191 Dec 21, 2022

The successor to nose, based on unittest2

Welcome to nose2 nose2 is the successor to nose. It's unittest with plugins. nose2 is a new project and does not support all of the features of nose.

736 Dec 16, 2022

Baseball Discord bot that can post up-to-date scores, lineups, and home runs.

Sunny Day Discord Bot Baseball Discord bot that can post up-to-date scores, lineups, and home runs. Uses webscraping techniques to scrape baseball dat

1 Jun 20, 2022

A simple tool to test internet stability.

pingtest Description A personal project for testing internet stability, intended for use in Linux and Windows.

0 Oct 17, 2021

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries.

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries. With this f

1 Jul 11, 2022

An AWS Pentesting tool that lets you use one-liner commands to backdoor an AWS account's resources with a rogue AWS account - or share the resources with the entire internet 😈

276 Mar 03, 2021