devwiki:python

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
devwiki:python [2021/12/02 10:40] – [Python 3 Changes and How to Make 2.7 Code Works] yingdevwiki:python [2024/03/25 06:36] (current) – [Python and interaction with other API] ying
Line 1: Line 1:
 +
 +
 +====== Modern Python Practice ======
 +
 +  * nowaday, now one use python 2.x anymore, use python 3 standard, use this
 +    * use pathlib instead of os.path
 +    * use f"{keyword}" instead of "{0}".format(keyword), unless it is easier
 +
 +
 +====== Online Python run ======
 +
 +  * https://jupyter.org/try-jupyter/retro/notebooks/?path=notebooks/Intro.ipynb
 +  * to install locally: <code>pip3 install --upgrade pip</code>
 +
 +====== Install related ======
 +===== install Python3 on mac =====
 +
 +  * Mac has its own python install, very old, only for basic usage
 +  * to install Python3 from official python website, 
 +    - download python 3 for mac installer
 +    - install as usual and done
 +    - run python3 in cmd
 +    - then it pops: "python3" command requires the "the commandline developer tools" to install, click install
 +    - if you already install XCode (the developer suite for mac), then no need; if not, you can just install the smaller "Command line tools" with your Apple account
 +    - go Apple developer downloads page, search command line tools for xcode, download that and install
 +
 +===== Manage and handle multiple versions of python case =====
 +
 +Situation:
 +  * sometimes, you may have to keep both python 2.7, python 3.x
 +  * and sometimes you want to try python with AI, and AI need to install tons of AI related python modules
 +  * sometimes you just to keep python module in the min form to test something
 +
 +Solution:
 +  * for those case, when install python, never add python to system path
 +  * just download and make python folder a portable folder (you can install and copy the python whole folder to other machine as a portable app, better zip to copy, faster)
 +  * then your computer cmd never know where is python and won't random get your python from somewhere
 +  * then, dynamically set python into python in cmd session or batch file, or use full path to python.exe to run your script
 +    * session based method (win as example) <code dos>SET PATH=D:\App\Python;D:\App_Dev\PortableGit\bin;%PATH%</code>
 +    * batch file method <code dos test.batch>
 +@echo off
 +SetLocal EnableDelayedExpansion
 +set CustPython=D:\App\Python\
 +call !CustPython!python.exe my_script.py
 +</code>
  
  
Line 25: Line 70:
     import importlib     import importlib
     reload = importlib.reload # add reload     reload = importlib.reload # add reload
-    raw_input = input+    raw_input = input # add raw_input 
 +    xrange = range # range 
 +    long = int # int 
 +    unicode = str # str
 </code> </code>
 +
 +  * integer division difference in py3 vs py2, a/b
 +    * ref: https://blog.finxter.com/how-to-divide-two-integers-in-python/
 +    * for safe, use <code>35/60 = 0 # py2
 +35//60  = 0 # py2 , always use // for calculate minutes and seconds
 +
 +35/60 = 0.583 #py3
 +35//60 = 0 # py3
 +</code>
 +
 ====== Python basics ====== ====== Python basics ======
  
Line 87: Line 145:
 ===== OS system operation ===== ===== OS system operation =====
  
-  * get user folder <code python>os.path.expanduser('~')</code>+  * get user/home folder <code python> 
 +os.path.expanduser('~'# return home directory, in Maya it return document, in windows, it return user dir 
 +os.getenv("HOME") # return home directory 
 +os.getenv("USERPROFILE") # return user profile directory 
 +</code>
  
   * make python code able to print unicode directly without .encode('utf-8') for text <code python>import sys   * make python code able to print unicode directly without .encode('utf-8') for text <code python>import sys
Line 106: Line 168:
   * get class name <code python>self.__class__.__name__</code>   * get class name <code python>self.__class__.__name__</code>
   * get basename of folder and file <code python>os.path.basename(os.path.normpath('/folderA/folderB/folderC/folderD/')) # 'folderD'</code>   * get basename of folder and file <code python>os.path.basename(os.path.normpath('/folderA/folderB/folderC/folderD/')) # 'folderD'</code>
 +  * get file name without extension <code python>
 +file_name = os.path.basename(file_path).rsplit('.',1)[0]
 +import pathlib
 +file_name_2 = pathlib.Path(file_path).stem
 +</code>
   * get expanded and un-symbolic path <code python>   * get expanded and un-symbolic path <code python>
 os.path.abspath(__file__) os.path.abspath(__file__)
Line 426: Line 493:
 </code> </code>
  
 +  * to use non-ascii character in python file, you need to put encoding in first line of code, or you will get "Non-ASCII character '\xe2' in file" error <code python>
 +# coding: utf-8
 +my_special_string = u"好"
 +print(u"好")
 +</code>
 +  * remove extra space from a string, also remove end single space<code python>
 +tidy_name = ' '.join(old_name.split())
 +
 +# just remove extra space
 +tidy_name = re.sub(' +', ' ', old_name) # only space, need strip() to clear end space
 +tidy_name = re.sub('\s{2,}', ' ', old_name) # space, tab, new line, need strip() to clear end space
 +</code>
 =====  control flow operation ===== =====  control flow operation =====
  
Line 447: Line 526:
 not x not x
 </code> </code>
 +  * try operation, note try is not a block, variable is same scope before try <code python> 
 +try: 
 +    val = test(A) 
 +except: 
 +    pass 
 +# ref: https://teamtreehouse.com/community/scope-of-a-variable   
 +</code>
 ===== data operation ===== ===== data operation =====
  
Line 584: Line 669:
 list=[1,2,3,4,5,6] list=[1,2,3,4,5,6]
 seq=list[::-2] # [6,4,2] seq=list[::-2] # [6,4,2]
 +
 +# list to pair list
 +info = ['name','John', 'age', '17', 'address', 'abc']
 +info_pair = zip(info[::2], info[1::2])
 +# [('name', 'John'), ('age', '17'), ('address', 'abc')] 
  
 # list flatten, and remove duplicates # list flatten, and remove duplicates
Line 854: Line 944:
 except OSError: except OSError:
     pass     pass
 +    
 +# method 3: all in one check and create if needed, like method 2 but not need try and except since it will be ok for exist
 +os.makedirs(os.path.dirname(output_path), exist_ok=True)
 </code> </code>
   * write python data<code python>   * write python data<code python>
Line 902: Line 995:
         os.mkdir(to_mkdir)         os.mkdir(to_mkdir)
 os.startfile(tgtPath) # show result folder os.startfile(tgtPath) # show result folder
 +</code>
 +
 +===== thread operation =====
 +
 +  * run a process or python service/app into another thread, so it wont block current interaction <code python>
 +def app(opt):
 +    if opt == 'Noter':
 +        import threading
 +        th = threading.Thread(target = run_noter)
 +        th.start()
 +def run_noter():
 +    import Noter
 +    Noter.main()
 </code> </code>
  
Line 912: Line 1018:
 REM make sure that wheel is in 32/64bit matching your python 32/64bit version REM make sure that wheel is in 32/64bit matching your python 32/64bit version
 pip install C:/some-dir/some-file.whl pip install C:/some-dir/some-file.whl
 +</code>
 +  * How to install extra module into seperate directory <code>
 +python.exe -m pip install --target=D:\my_path\plot_lib matplotlib
 </code> </code>
   * How to install extra modules manually   * How to install extra modules manually
Line 1037: Line 1146:
 </code> </code>
  
 +  * data hashing, like compress data into a string, good for big data compare and record
 +    * hash (built-in, not need import) 64bit, take a object and get a int for quick compare
 +    * hashlib (built-in, need import) 128bit+ and more option like md5, sha for more secure and digital signiture
 ==== system info modules ==== ==== system info modules ====
  
Line 1087: Line 1199:
 info=subprocess.Popen('tasklist.exe /FO CSV /FI "IMAGENAME eq {0}"'.format(appName),stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE) info=subprocess.Popen('tasklist.exe /FO CSV /FI "IMAGENAME eq {0}"'.format(appName),stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
 out, err = info.communicate() out, err = info.communicate()
 +#out = str(out) # for py3 byte convert
 +out = out.decode('ascii')
 result = [ x for x in out.replace('"','').split('\r\n') if x !='' ] result = [ x for x in out.replace('"','').split('\r\n') if x !='' ]
 if len(result)>1 and result[1].startswith(appName): if len(result)>1 and result[1].startswith(appName):
Line 1130: Line 1244:
 subprocess.Popen([r'D:\Dev\Pipeline\PyDesktop\CompFolder\CompFolder.bat'], env=env2) subprocess.Popen([r'D:\Dev\Pipeline\PyDesktop\CompFolder\CompFolder.bat'], env=env2)
 </code> </code>
 +    * subprocess a console app and get return, and optionally hide the pop up console when call <code python>
 +# hide popup console for win platform only
 +startupinfo = subprocess.STARTUPINFO()
 +startupinfo.dwFlags |= 1 #subprocess.STARTF_USESHOWWINDOW
 +# startupinfo.wShowWindow = subprocess.SW_HIDE #(optional)
  
 +info=subprocess.Popen( '{0}'.format(app_path),stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE,  startupinfo=startupinfo)
 +out, err = info.communicate()
 +out = out.decode('ascii') #str(out) # for py3 byte convert
 +</code>
 ==== string related ==== ==== string related ====
  
Line 1214: Line 1337:
  
   * 3rd party module, html get post request response operation library   * 3rd party module, html get post request response operation library
 +  * header list: https://deviceatlas.com/blog/list-of-user-agent-strings
  
   * **Single Get web link content** <code python>   * **Single Get web link content** <code python>
Line 1240: Line 1364:
 result.content # is html in bytes result.content # is html in bytes
 result.text # is html in unicode text result.text # is html in unicode text
 +</code>
 +  * disable non secure cert warning <code python>requests.packages.urllib3.disable_warnings(requests.packages.urllib3.exceptions.InsecureRequestWarning)</code>
 +===== PIL and Pillow=====
 +
 +  * PIL, Python Imaging Library (Fork) =  Pillow, install the lib <code>python -m pip install Pillow</code>
 +  * python code <code python>
 +from PIL import Image, ImageDraw, ImageOps
 </code> </code>
  
Line 1245: Line 1376:
  
   * a html info extractor, best works with another parser module "lxml" to get partial info   * a html info extractor, best works with another parser module "lxml" to get partial info
 +  * ref: https://beautiful-soup-4.readthedocs.io/en/latest/#installing-a-parser 
 +    * parser is a html interprater, as not all html is clean format and tidy, some parser can auto fix some html tag miss match.
   * example code <code python>   * example code <code python>
 from bs4 import BeautifulSoup from bs4 import BeautifulSoup
Line 1445: Line 1577:
 def get_win_order_all(): def get_win_order_all():
     # ctype: grab folder windows in z order, return [(title, class, [x,y,w,h], hwnd int)]     # ctype: grab folder windows in z order, return [(title, class, [x,y,w,h], hwnd int)]
 +    from ctypes import wintypes
     order_list = []     order_list = []
     result_list = []     result_list = []
Line 1457: Line 1590:
     r=ctypes.wintypes.RECT()     r=ctypes.wintypes.RECT()
     ctypes.windll.user32.GetWindowRect(top,ctypes.byref(r))     ctypes.windll.user32.GetWindowRect(top,ctypes.byref(r))
-    result_list.append( [buff.value, class_name.value, [r.left,r.top,r.right-r.left,r.bottom-r.top], top ])+    # class_name.value in py3 is byte, while py2 is string 
 +    if isinstance(class_name.value, str): 
 +        result_list.append( [buff.value, class_name.value, [r.left,r.top,r.right-r.left,r.bottom-r.top], top ]) 
 +    else: 
 +        result_list.append( [buff.value, class_name.value.decode('ascii'), [r.left,r.top,r.right-r.left,r.bottom-r.top], top ])
     order_list.append(top)     order_list.append(top)
     while True:     while True:
Line 1470: Line 1607:
         r=ctypes.wintypes.RECT()         r=ctypes.wintypes.RECT()
         ctypes.windll.user32.GetWindowRect(next,ctypes.byref(r))         ctypes.windll.user32.GetWindowRect(next,ctypes.byref(r))
-        result_list.append( [buff.value, class_name.value, [r.left,r.top,r.right-r.left,r.bottom-r.top], next] )+        # class_name.value in py3 is byte, while py2 is string 
 +        if isinstance(class_name.value, str): 
 +            result_list.append( [buff.value, class_name.value, [r.left,r.top,r.right-r.left,r.bottom-r.top], next] ) 
 +        else: 
 +            result_list.append( [buff.value, class_name.value.decode('ascii'), [r.left,r.top,r.right-r.left,r.bottom-r.top], next] )
         order_list.append(next)         order_list.append(next)
     return result_list     return result_list
Line 1495: Line 1636:
             class_name = ctypes.create_string_buffer(200)             class_name = ctypes.create_string_buffer(200)
             ctypes.windll.user32.GetClassNameA(hwnd, ctypes.byref(class_name), 200)             ctypes.windll.user32.GetClassNameA(hwnd, ctypes.byref(class_name), 200)
-            if class_name.value == 'CabinetWClass': +            t_class_name = class_name.value 
-                window_list.append( [ buff.value, class_name.value, hwnd ])+            if not isinstance(t_class_name, str): 
 +                t_class_name = class_name.value.decode('ascii') # py3 fix 
 +            if t_class_name == 'CabinetWClass': 
 +                window_list.append( [ buff.value, t_class_name, hwnd ])
         return True         return True
     procObj = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int))     procObj = ctypes.WINFUNCTYPE(ctypes.c_bool, ctypes.POINTER(ctypes.c_int), ctypes.POINTER(ctypes.c_int))
Line 1554: Line 1698:
     class_name = ctypes.create_string_buffer(200)     class_name = ctypes.create_string_buffer(200)
     ctypes.windll.user32.GetClassNameA(hwnd, ctypes.byref(class_name), 200)     ctypes.windll.user32.GetClassNameA(hwnd, ctypes.byref(class_name), 200)
-    return class_name.value+    t_class_name = class_name.value 
 +    if not isinstance(t_class_name, str): 
 +        t_class_name = class_name.value.decode('ascii') # py3 fix 
 +    return t_class_name
 def is_win_top(hwnd): def is_win_top(hwnd):
     top_hwnd = ctypes.windll.user32.GetTopWindow(None)     top_hwnd = ctypes.windll.user32.GetTopWindow(None)
Line 1744: Line 1891:
  
   * a python web server, install by <code>python -m pip install django</code>   * a python web server, install by <code>python -m pip install django</code>
 +
 +===== openpyxl=====
 +
 +  * a new excel reader, xlsx format, not old xls format, since now everyone use new excel version
 +  * read excel as dictionary (1st key, rest data) <code python>
 +    def excel2Dict(filename):
 +        # import openpyxl
 +        # max_row = sheet.max_row
 +        # max_column = sheet.max_column
 +        wb = openpyxl.load_workbook(filename=filename, read_only=True)
 +        records = []
 +        for sheet in wb.worksheets:
 +            count = 0
 +            keys = []
 +            
 +            for row in sheet.rows:
 +                temp = []
 +                for cell in row:
 +                    if count == 0:
 +                        keys.append(str(cell.value))
 +                    else:
 +                        temp.append(str(cell.value))
 +                if count > 0:
 +                    records.append( dict(zip(keys, temp)) ) # 
 +                else:
 +                    count += 1
 +        return records
 +</code>
 +  * read excel as row data wo empty cell. <code python>
 +    def excel2Data(filename):
 +        # import openpyxl
 +        wb = openpyxl.load_workbook(filename=filename, read_only=True, data_only=True)
 +        sheet_data = []
 +        for sheet in wb.worksheets:
 +            table_data = []
 +            for row in sheet.values:
 +                row_data = []
 +                for cell in row:
 +                    if cell is not None:
 +                        row_data.append(cell)
 +                if len(row_data)>0:
 +                    table_data.append(row_data)
 +            sheet_data.append(table_data)
 +        return sheet_data
 +</code>
 ====== Python by Reference or by Value ====== ====== Python by Reference or by Value ======
  
Line 2145: Line 2337:
     * when first run, allow Access folder right.     * when first run, allow Access folder right.
   * to make command extension file runable, do bash for the command extension launcher <code bash>chmod 755 App_V9.command</code>   * to make command extension file runable, do bash for the command extension launcher <code bash>chmod 755 App_V9.command</code>
-====== Python Cool Tricks ======+====== Python Cool Tricks and Notes ======
  
   * start HTML and FILE server in current directory <code dos>python -m SimpleHTTPServer 8181</code>   * start HTML and FILE server in current directory <code dos>python -m SimpleHTTPServer 8181</code>
 +  * decimal vs float for real life accounting usage:
 +    * decimal wont have those float annoying issue with tons of 0s after .
 +    * decimal deal number like human do instead of infinite computing accuracy
 +    * https://towardsdev.com/python-tips-decimal-vs-float-72fe72fb7086
 +    * https://www.laac.dev/blog/float-vs-decimal-python/
 +    * <code>from decimal import Decimal; a = Decimal(0.01)</code>
 +
 +
 +  * python and whatsapp msg
 +    * ref: https://iq.opengenus.org/python-script-to-send-whatsapp-message/
 +    * **pywhatkit**: use web client to automate msg sending, make sure it on main screen (multi screen case) 
 +      * because it use automation, so interaction with browser can be buggy, no quick repeat sending (since automate like human action, need wait response of other app update), send group chat may send to wrong group if you also interact with browser
 +      * example<code python>
 +import pywhatkit
 +pywhatkit.sendwhatmsg_instantly("+88123456",auto_url,10,True,3) 
 +# wait 10s to ui response, close tab true after 3s, 
 +# you need to close tab as whatsapp web only allow 1 window active
 +</code>
 +    * **selenium**: use selenium browser automate driver to operate, since it is linking to a dedicated browser with browser api, it is more reliable to send fast msg and maintain connection, no user interaction affection
 +      * example setup and login manually for first time <code python>
 +my_drive = None
 +def drive_setup():
 +    from selenium import webdriver
 +    from selenium.webdriver.common.keys import Keys
 +    from selenium.webdriver.common.by import By
 +    
 +    my_drive = webdriver.Chrome()
 +    phone_no = "+881234567" # target phone or your own phone as target
 +    my_drive.get(f'https://web.whatsapp.com/send?phone={phone_no}&text=')
 +
 +def drive_msg(in_text):
 +    from selenium.webdriver.common.by import By
 +    if my_drive is None:
 +        return
 +    else:
 +        chat_box = my_drive.find_element(by=By.XPATH, value='/html/body/div[1]/div/div/div[4]/div/footer/div[1]/div/span[2]/div/div[2]/div[1]/div/div/p')
 +        chat_box.send_keys(in_text)
 +        send_button = my_drive.find_element(by=By.CSS_SELECTOR, value='[data-icon="send"]')
 +        send_button.click()
 +</code>
 +
 +====== Python and interaction with other API ======
 +
 +Telegram
 +  - search @BotFather on telegram
 +    * manual: https://core.telegram.org/bots/api
 +  - in the chat, type /newbot, then type your bot "botname", then type your bot "botusername_bot"
 +  - now, your bot api key will show up, copy and save it
 +  - now search your bot "botusername_bot" to start chat with it
 +  - in python, use requests to get the chat from the bot, try one more time if no result <code python>
 +import requests
 +the_key = "api_key"
 +url = "https://api.telegram.org/bot{0}/getUpdates".format(the_key)
 +response = requests.get(url)
 +result = response.json()
 +print(result)
 +</code>
 +  - once you got result in json, you will find the chat id. <code python>result['result']
 +# the list of all chat spec your bot received, yours is likely the first one.
 +my_chat_info = result['result']
 +my_msg_info = my_chat_info[0]['message']
 +# it got info of: chat, date, from, message_id, text
 +my_chat_id = my_msg_info['chat']['id']
 +</code>
 +  - to send a text to that chat from your bot side <code python>
 +reply_text = 'hello world'
 +url = "https://api.telegram.org/bot{key}/sendMessage?chat_id={id}&text={reply}".format(**{'key':the_key, 'id':my_chat_id,'reply':reply_text})
 +response = requests.get(url)
 +</code>
 +  - now, you should have got the reply from your bot <code python>
 +result = response.json()
 +send_state = result['ok']
 +send_result = result['result']
 +# chat (first_name, id, type); date:int; message_id; text
 +</code>
 +
 +====== Python and Networking related ======
 +
 +  * above is python interact with a REST API server, you can also use python to build your own REST API server to handle requests from other application, even other application over the network and internet.
 +  * implement REST API into python, you may need to use some python module like Flash, FastAPI, then you start another thread from your python application to run your REST API server.
 +  * while using Sockets method may be simpler if you just want to send some simple message between applications, as REST API, you need write all the API URL end point response.
 +  * using REST API allow your application to talk cross different application and difference device, and provide a safe gate to your important data handled by your application.
  
  • devwiki/python.1638441645.txt.gz
  • Last modified: 2021/12/02 10:40
  • by ying