API to parse tibia.com content into python objects.

Last update: Oct 31, 2022

Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

Features:

Converts data into well-structured Python objects.
Type consistent attributes.
All objects can be converted to JSON strings.
Can be used with any networking library.
Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments

DisplayItems attribute name is returning None
I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

Here is my code snippet:

response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, fetch_items=True)) details : AuctionDetails = response.data items : List[DisplayItem] = details.items.entries print(items)

The print of the returned list:

[<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?
opened by guriosam 6
Problem with the comma when there are more than 1000 players online.

Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

opened by Pixelao 2
tibiapy.HousesSection.from_content is broken

Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

opened by aaronga97 1
v2.4.0
Changes

Added support for multiple houses per character. Accessible on Character.houses field.

Character.house is now deprecated. It will contain the character's first house or None.
opened by Galarzaa90 1
v1.1.0
Changes

Parsing Highscores from Tibia.com and TibiaData.

Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.

Added type hints to many variables and methods.
opened by Galarzaa90 1
Parse character position

Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

Not to be confused with the position field in Account Information.

Also, these characters show Cipsoft Member in their status when listed as "Other Characters".
enhancement

opened by Galarzaa90 1
Add HouseStatus and Sex enums

Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

opened by Tschis 1
Refactor guild.py and unit tests
Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;

Add new resource and unit tests for best case scenario on each section;
opened by Tschis 1
Parse character titles and main character information

Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

opened by Galarzaa90 0
v1.0.0
Changes

Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.

Added support for parsing Houses, House lists, World and World list

Added support for many missing attributes in Character and Guilds.

All objects are now serializable to JSON strings.
opened by Galarzaa90 0
Add /tests as a package

As seen here, this might help running the tests from the root folder.

Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

opened by Tschis 0

Releases(v5.5.2)

v5.5.2(Sep 3, 2022)
Changes

Fixed Houses section not parsing due to a change in the filters table.

Fixed status parameter not generating the correct URL in the houses section.

Source code(tar.gz)
Source code(zip)
v5.5.1(Aug 2, 2022)
Changes

Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:

Highscores

News

Forums

Spells

Source code(tar.gz)
Source code(zip)
v5.5.0(Jul 27, 2022)
Changes

Added BOSS_POINTS to Category in highscores.

Source code(tar.gz)
Source code(zip)
v5.4.0(Jul 23, 2022)
Changes

Added boss_points and bosstiary_progress to auctions.

Source code(tar.gz)
Source code(zip)
v5.3.0(Jul 23, 2022)
Changes

Added support for Boostable Bosses.

Source code(tar.gz)
Source code(zip)
v5.2.1(Mar 1, 2022)
Changes

Fixed bug in news ticker with a div tag failing to parse.

Updated code to detect the discussion thread link.

Source code(tar.gz)
Source code(zip)
v5.2.0(Dec 31, 2021)
Changes

Added exalted_dust and exalted_dust_limit attributes to auctions.

Source code(tar.gz)
Source code(zip)
v5.1.0(Sep 16, 2021)
Changes

Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.

Properly handle deaths caused by summons of traded characters.

Source code(tar.gz)
Source code(zip)
v5.0.1.post0(Aug 26, 2021)

Source code(tar.gz)
Source code(zip)
v5.0.1(Aug 26, 2021)
Changes

Fixed many sections not being parsed correctly due to changes to Tibia.com.

Houses list

News list

Spells section

Forums section

Fixed character's houses failing to parse due to a bug in the display in Tibia.com.

Temporarily disabling this attribute.

Source code(tar.gz)
Source code(zip)
v5.0.1a1(Aug 24, 2021)
Prelease to address part of the Tibia.com changes.

Changes

Fixed character's houses failing to parse due to a bug in the display in Tibia.com.

Temporarily disabling this attribute.

Fixed house list not parsing correctly due to Tibia.com changes

Fixed news list not parsing correctly due to Tibia.com changes

Pending

Forums section

Spells section

Source code(tar.gz)
Source code(zip)
v5.0.0(Aug 7, 2021)
Changes

Added parsing for Tibia Drome leaderboards, new Leaderboard class.

Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.

New Client method: fetch_leaderboards.

Added parsing for Spells library, new SpellsSection class.

Auxiliary classes Spell and SpellEntry.

New Client methods: fetch_spell and fetch_spells

Fix last page of highscores having 0 as page value.

Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.

Note that if the test website has changes, parsing might not be possible.

Internal URL attributes might still point to the regular website.

New HousesSection class, including the house filtering attributes.

New NewsArchive class, including the news filtering attributes.

New GuildsSection class, to replace the lists of GuildEntry

Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.

Fixed bug with other characters not being parsed.

Added traded attribute to LastPost class.

Added thread_starter_traded attribute to ThreadEntry

Breaking Changes:

Python 3.7 or higher is now required.

House classes no longer have get_list_url and list_from_content methods.

Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.

ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.

ListedHouse class renamed to HouseEntry.

Removed deprecated property AuctionFilters.item.

Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.

ListedNews class renamed to NewsEntry.

News and NewsEntry no longer have a get_list_url method.

ListedBoard class renamed to BoardEntry.

ListedThread class renamed to ThreadEntry.

ListedAnnouncement class renamed to AnnouncementEntry.

ListedWorld class renamed to WorldEntry.

ListedAuction class renamed to AuctionEntry.

AuctionDetails class renamed to Auction.

ListedGuild class renamed to GuildEntry.

ListedTournament class renamed to TournamentEntry.

Creature class renamed to CreatureEntry.

CreatureDetail class renamed to Creature.

Guild and GuildEntry class no longer have a get_list_url method.

Renamed begin_date parameter to start_date in fetch_news_archive.

Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.

CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.

Source code(tar.gz)
Source code(zip)
v4.1.7(Jun 30, 2021)
Changes

Fixed titles being parsed as part of the name for guild members with symbols in their name.

Source code(tar.gz)
Source code(zip)
v4.1.6(Jun 28, 2021)
Changes

Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.

Source code(tar.gz)
Source code(zip)
v4.1.5(Jun 26, 2021)
Changes

Fixed parsing bug on characters, returning an incorrect exception when a character doesn't exist.

Source code(tar.gz)
Source code(zip)
v4.1.4(Jun 17, 2021)
Changes

Fixed worlds not being parsed correctly due to tournament worlds order changing.

Source code(tar.gz)
Source code(zip)
v4.1.3(May 12, 2021)
Changes

Fixed house transfer date not being parsed properly.

Source code(tar.gz)
Source code(zip)
v4.1.2(Apr 27, 2021)
Changes

Fixed parsing errors for forum posts that contained a copy of the signature separator in the signature.

Source code(tar.gz)
Source code(zip)
v4.1.1(Apr 19, 2021)
Changes

Fixed bug with extraneous character in some item descriptions, causing auction to give a parsing error.

Source code(tar.gz)
Source code(zip)
v4.1.0(Mar 30, 2021)
Changes

Added prey_wildcards attribute to AuctionDetails.

Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.

Source code(tar.gz)
Source code(zip)
v4.0.0(Mar 11, 2021)
Changes

Breaking change: Removed BoostedCreature class, replaced by Creature class.

Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.

Added parsing and fetching for the Creature library section.

Added CreatureSection, Creature, and CreatureDetail classes.

Added traded attribute to ForumAuthor. Indicates if the author was a traded character.

Previously, it would mark the author as a deleted character and its name would include (traded).

Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.

Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.

battleye_protected is now a property instead of an attribute.

Added YELLOW and GREEN aliases to all BattlEye related enums.

Fixed wrong timezone being used for forum related dates.

Source code(tar.gz)
Source code(zip)
v3.7.1(Feb 16, 2021)
Changes

Adjusted highscores parsing for upcoming Tibia.com changes.

Source code(tar.gz)
Source code(zip)
v.3.7.0(Feb 9, 2021)
Changes

Parse familiars from auctions

Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.

Results count in bazaar pages are now properly parsed when there are comma thousand separators.

Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)

Source code(tar.gz)
Source code(zip)
v3.6.5(Jan 28, 2021)
Changes

Fixed auction history parsing breaking due to the cookie consent dialog.

Source code(tar.gz)
Source code(zip)
v3.6.4(Jan 26, 2021)
Changes

Fixed world list parsing breaking due to the cookie consent dialog.

Source code(tar.gz)
Source code(zip)
v3.6.3(Jan 15, 2021)
Changes

Fixed bug in guild names being parsed with Non-Breaking spaces instead of a regular space.

Source code(tar.gz)
Source code(zip)
v3.6.2(Jan 2, 2021)
Changes

Fixed bug in Event Calendar parsing.

Source code(tar.gz)
Source code(zip)
v3.6.1(Dec 28, 2020)
Changes

Fixed guild information being parsed incorrectly for characters in guilds containing "of the" in their name.

Source code(tar.gz)
Source code(zip)
v3.6.0(Dec 12, 2020)
Changes

Added support for the new filtering options in Highscores

Added battleye_filter and pvp_types_filter attributes.

Added get_page_url() instance method to Highscores class.

Added previous_page_url and next_page_url properties.

Source code(tar.gz)
Source code(zip)
v3.5.7(Dec 4, 2020)
Changes

Fixed bug in Event Calendar parsing.

Source code(tar.gz)
Source code(zip)

Owner

Allan Galarza

Mechatronic Engineer

GitHub Repository https://tibiapy.readthedocs.io/

This script is intended to crawl license information of repositories through the GitHub API.

GithubLicenseCrawler This script is intended to crawl license information of repositories through the GitHub API. Taking a csv file with requirements.

4 Oct 25, 2022

此脚本为 python 脚本,实现原理为利用 selenium 定位相关元素,再配合点击事件完成浏览器的自动化.

5 Nov 19, 2021

🐞 Douban Movie / Douban Book Scarpy

Python3-based Douban Movie/Douban Book Scarpy crawler for cover downloading + data crawling + review entry.

1 Dec 03, 2022

Twitter Eye is a Twitter Information Gathering Tool With Twitter Eye

Twitter Eye is a Twitter Information Gathering Tool With Twitter Eye, you can search with various keywords and usernames on Twitter.

19 Dec 12, 2022

对于有验证码的站点爆破，用于安全合法测试

使用方法 python3 main.py + 配置好的文件 python3 main.py Verify.json python3 main.py NoVerify.json 以上分别对应有验证码的demo和无验证码的demo Tips: 你可以以域名作为配置文件名字加载：python3 main

47 Nov 09, 2022

Scraping news from Ucsal portal with Scrapy.

NewsScraping Esse é um projeto de raspagem das últimas noticias, de 2021, do portal da universidade Ucsal http://noosfero.ucsal.br/institucional Tecno

0 Sep 30, 2021

jd_maotai rpa 基于selenium驱动的jd抢购rpa机器人

jd_maotai rpa 基于selenium驱动的jd抢购rpa机器人, 照顾我们这样的马大哈, 不会忘记抢购了, 祝大家过年都能喝上茅台. 特别声明: 本仓库发布的jd_maotai_rpa项目定义为自动化rpa项目, 是用于防止忘记参与jd茅台的活动(由于本人时常忘记), 而不是为了秒杀和抢

35 Nov 18, 2022

河南工业大学完美校园自动校外打卡

HAUT-checkin 河南工业大学自动校外打卡由于github actions存在明显延迟，建议直接使用腾讯云函数特点多人打卡使用简单，仅需账号密码以及用于微信推送的uid 自动获取上一次打卡信息用于打卡向所有成员微信单独推送打卡状态完美校园服务器繁忙时造成打卡失败会自动重新打卡

36 Oct 27, 2022

This is a simple website crawler which asks for a website link from the user to crawl and find specific data from the given website address.

1 Jan 10, 2022

API to parse tibia.com content into python objects.

Related tags

Overview

Tibia.py

Installing

Usage

Documentation

Comments

Changes

Changes

Changes

Releases(v5.5.2)

v5.5.2(Sep 3, 2022)

Changes

v5.5.1(Aug 2, 2022)

Changes

v5.5.0(Jul 27, 2022)

Changes

v5.4.0(Jul 23, 2022)

Changes

v5.3.0(Jul 23, 2022)

Changes

v5.2.1(Mar 1, 2022)

Changes

v5.2.0(Dec 31, 2021)

Changes

v5.1.0(Sep 16, 2021)

Changes

v5.0.1.post0(Aug 26, 2021)

v5.0.1(Aug 26, 2021)

Changes

v5.0.1a1(Aug 24, 2021)

Changes

Pending

v5.0.0(Aug 7, 2021)

Changes

v4.1.7(Jun 30, 2021)

Changes

v4.1.6(Jun 28, 2021)

Changes

v4.1.5(Jun 26, 2021)

Changes

v4.1.4(Jun 17, 2021)

Changes

v4.1.3(May 12, 2021)

Changes

v4.1.2(Apr 27, 2021)

Changes

v4.1.1(Apr 19, 2021)

Changes

v4.1.0(Mar 30, 2021)

Changes

v4.0.0(Mar 11, 2021)

Changes

v3.7.1(Feb 16, 2021)

Changes

v.3.7.0(Feb 9, 2021)

Changes

v3.6.5(Jan 28, 2021)

Changes

v3.6.4(Jan 26, 2021)

Changes

v3.6.3(Jan 15, 2021)

Changes

v3.6.2(Jan 2, 2021)

Changes

v3.6.1(Dec 28, 2020)

Changes

v3.6.0(Dec 12, 2020)

Changes

v3.5.7(Dec 4, 2020)

Changes

Owner

Allan Galarza

This script is intended to crawl license information of repositories through the GitHub API.

此脚本为 python 脚本,实现原理为利用 selenium 定位相关元素,再配合点击事件完成浏览器的自动化.

🐞 Douban Movie / Douban Book Scarpy

Twitter Eye is a Twitter Information Gathering Tool With Twitter Eye

对于有验证码的站点爆破，用于安全合法测试

Scraping news from Ucsal portal with Scrapy.

河南工业大学完美校园自动校外打卡