This speeds up PyCharm's package index processes and avoids CPU & memory overloading

Overview
Avoid PyCharm Overloading
=========================

This speeds up PyCharm's package index processes and avoids CPU & memory
overloading.

On my machine (4c/32GB) with 5 conda environments and 10 projects, this reduces
PyCharm's package scanning from 15 minutes to 45 seconds.

Why?
----

When PyCharm has multiple conda environments, it will sometimes launch concurrent
package scanning across those environments. To do so, it uses the `conda` cli,
and a tool called conda_packaging_tool.py, available from the JetBrains
intellij-community github repository (see below).

Unfortunately, these tools do not behave well when launched in parallel, as
they tend to overload CPU & memory in this case.

How?
----

To avoid stalling these processes, we modify their sources as follows:

1. modify the conda cli

    $ conda activate base
    $ pip install runfast
    $ nano `which conda`

    Modify the main section:

     if __name__ == '__main__':
        from conda.cli import main
        from runfast import cached
        cached(main)

2. modify the conda_packaging_tool.py

    if __name__ == '__main__':
        from runfast import cached
        cached(main)

How does this work?
-------------------

Two simple steps:

1. When called in parallel, only one process is allowed to proceed.
2. `runfast.cached` caches the output of these tools (stdout, stderr) for 1 minute,
   given the same command line parameters

That is, when PyCharm launches 5 package scanning (`conda list`) commands, only
one of them will run immediately. If some of the scans are for the same environment,
only one of them will actually run, while the others simply return the same output.

How to clear the cache or avoid caching at all?

    $ export RUNFAST_NOCACHE=1

Testing
-------

a) conda packaging tool

    # first time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m13.271s
    user    0m9.631s
    sys     0m1.936s

    # second time
    $ time python /opt/pycharm-community-2021.2.1/plugins/python-ce/helpers/conda_packaging_tool.py
    real    0m3.159s
    user    0m0.139s
    sys     0m1.298s

b) conda cli

   # first time
   $ time conda list -p /path/to/env
   real    0m7.985s
   user    0m7.883s
   sys     0m0.087s

   # second time
   real    0m0.152s
   user    0m0.116s
   sys     0m0.032s

References
----------
* https://github.com/JetBrains/intellij-community/blob/master/python/helpers/conda_packaging_tool.py
* https://github.com/conda/conda/blob/33a142c16530fcdada6c377486f1c1a385738a96/conda/core/index.py#L53
Owner
tinkerer | software engineer | data scientist
Collection of script & resources for Foundry's Nuke software.

Author: Liam Collod. Collections of scripting stuff I wrote for Foundry's Nuke software. Utilisation You can have a look at the README.md file in each

Liam Collod 1 May 14, 2022
PyDateWaiter helps waiting special day & calculating remain days till that day with Python code.

PyDateWaiter (v.Beta) PyDateWaiter helps waiting special day(aniversary) & calculating remain days till that day with Python code. Made by wallga gith

wallga 1 Jan 14, 2022
Annotates sequences with Eggnog-mapper and hhblits against PDB70

Annotating "hypothetical" proteins with the PDB See config/ for configuration information. This workflow takes as input a set of protein sequences. It

1 Apr 05, 2022
DeDRM tools for ebooks

DeDRM_tools DeDRM tools for ebooks This is a fork of Apprentice Harper's version of the DeDRM tools. I've added some of the PRs that still haven't bee

2 Jan 10, 2022
A Classroom Engagement Platform

Project Introduction This is project introduction Setup Setting up Postgres This is the most tricky part when setting up the application. You will nee

Santosh Kumar Patro 1 Nov 18, 2021
Labspy06 With Python

Labspy06 Profil Nama : Nafal mumtaz fuadi Nim : 312110457 Kelas : T1.21.A.2 Latihan 1 Ubahlah kode dibawah ini menjadi fungsi menggunakan lambda impor

Mas Nafal 1 Dec 12, 2021
AKSWINPOSTINIT -- AKS Windows node post provisioning initialization

AKSWINPOSTINIT -- AKS Windows node post provisioning initialization Features This is a tool that provides one-time powershell script initilization for

Ping He 3 Nov 25, 2021
WorldsCollide - Final Fantasy VI Randomizer

FFVI Worlds Collide Worlds Collide is an open worlds randomizer for Final Fantas

8 Jun 13, 2022
A python script developed to process Windows memory images based on triage type.

Overview A python script developed to process Windows memory images based on triage type. Requirements Python3 Bulk Extractor Volatility2 with Communi

CrowdStrike 245 Nov 24, 2022
NFT-Image-Generator - Utility to generate a large collection of unique images

NFT-Image-Generator Utility for creating a generative art collection from suppli

Sem Moolenschot 60 Dec 15, 2022
From "fixed RAnDom CRashes" to "[FIX] Fixed random crashes."

Clean Commit From fixed RAnDom CRashes to [FIX] Fixed random crashes. Clean commit helps you by auto-formating your commits to make your repos better

Mathias 3 Dec 26, 2021
A small site to list shared directories

Nebula Server Directories This site can be used to list folder and subdirectories in your server : Python It's required to have Python 3.8 or more ins

Adrien J. 1 Dec 28, 2021
A small project of two newbies, who wanted to learn something about Python language programming, via fun way.

HaveFun A small project of two newbies, who wanted to learn something about Python language programming, via fun way. What's this project about? Well.

Patryk Sobczak 2 Nov 24, 2021
Expression interpreter written in Python

Calc Interpreter An interpreter modeled after a calculator implemented in Python 3. The program currently only supports basic mathematical expressions

1 Oct 17, 2021
A micro-service that can be extended to help in monitoring systems

A micro-service that can be extended to help in monitoring systems. Be extensible to be incorporated in any of the systems to facilitate timely interventions.

Peter Kagwe 1 Feb 06, 2022
Flexible constructor to create dynamic list of heterogeneous properties for some kind of entity

Flexible constructor to create dynamic list of heterogeneous properties for some kind of entity. This set of helpers useful to create properties like contacts or attributes for describe car/computer/

Django Stars 24 Jul 21, 2022
The purpose is to have a fairly simple python assignment that introduces the basic features and tools of python

This repository contains the code for the python introduction lab. The purpose is to have a fairly simple python assignment that introduces the basic

1 Jan 24, 2022
An example file showing a simple endpoints like a login/logout function and maybe some others.

Flask API Example An example project showing a simple endpoints like a login/logout function and maybe some others. How to use: Open up your IDE (or u

Kevin 1 Oct 27, 2021
《赛马娘》(ウマ娘: Pretty Derby)辅助 🐎🖥 基于 auto-derby 可视化操作/设置 启动器 一键包

ok-derby 《赛马娘》(ウマ娘: Pretty Derby)辅助 🐎 🖥 基于 auto-derby 可视化操作/设置 启动器 一键包 便捷,好用的 auto_derby 管理器! 功能 支持客户端 DMM (前台) 实验性 安卓 ADB 连接(后台)开发基于 1080x1920 分辨率

秋葉あんず 90 Jan 01, 2023
Howell County, Missouri, COVID-19 data and (unofficial) estimates

COVID-19 in Howell County, Missouri This repository contains the daily data files used to generate my COVID-19 dashboard for Howell County, Missouri,

Jonathan Thornton 0 Jun 18, 2022