Python commands Hugo Touchette Department of Mathematical Sciences Stellenbosch University Updated: 11 June 2020 --------------------------------------------------------- Features - Dynamic types (variable type can change) - No type declaration - Clean syntax, no end of line symbol - List comprehension - Extensive libraries (Numpy, SciPy, SciKit) - Methods oriented (mimicking postfix application) --------------------------------------------------------- Environments ipython --pylab Interactive python Basic libraries are included (Numpy, Matplotlib, etc.) python: Bare python command line python3: python v3 Running scripts: python file.py python3 file.py #!/usr/bin/env python #!/usr/bin/env python3 Google notebook: colaboratory or colab Anaconda Python (with IDLE Spyder) Jupyter = old IPython jupyter notebook jupyter lab (for fancy version) --------------------------------------------------------- Version compatibility - enable v3 features in v2 from __future__ import print_function from __future__ import division --------------------------------------------------------- Packages import [package] as [acronym] All functions available in namespace package.function All .py files and functions from .py files can be imported from [package] import [function1], [function2] Function is then called without package name from package import * Import all with given names Important packages: math log, log10, exp, sin, cos, asin, sinh e, pi, degrees, radians, fabs, pow cmath Complex maths numpy (as np) median, average, mean, std, var Many others .fft .linalg (LINPACK) .random Functions better than in math/cmath because listable matplotlib.pyplot (as plt) scipy Root finders, optimization, curve fitting, distributions, etc. Bigger vectors and matrices pylab = numpy, scipy and matplotlib seaborn (as sns) Nicer plots pandas (as pd) scikit-learn statsmodels keras Neural net API datetime networkx (as nx) --------------------------------------------------------- Package installation pip install package Should be v2 but doesn't work on my installation pip3 install package v3 python -m pip install package Force v2 install python3 -m pip install package v3 conda install For Conda installations sudo -H System wide install --------------------------------------------------------- Magic commands %timeit operation %%time %pwd %run file.py --------------------------------------------------------- General # Comments """ ... """ Pydoc/docstring comments: Assert condition, returns error if not met: assert condition, "Error message" Hash-bang or shebang: #!/usr/bin/env python #!/usr/bin/python --------------------------------------------------------- Variables and types Assignments not dynamic: x=3, y=x, x=5 doesn't change y Multiple assignments: x = y = 3 Dummy (void) assignments: x, _ = 1, 2 Tuple assignment: x, y = 3, 4 Swap: x, y = y, x Variables must start with letter Names are case sensitive Global and local variables can be handled - to avoid! Integer max = 2^31-1 Long Integer: with trailing L Float Complex: x=0.5+1.2j (no space or * with j) Real part: a.real or np.real(a) Imaginary part: a.imag or np.imag(a) Strings: '...' or "..." Python automatically converts types as necessary Examples: 1/2 -> 0 (v2 not v3) 3/2 -> 1.5 (v3) 1.0/2.0 -> 0.5 1.0/2 -> 0.5 Boolean: True and False Type conversion: float() int() complex() bool() str() type(x) Returns type of x --------------------------------------------------------- Operations + - * Multiplication or concatenation for lists / Division; int division if used on ints // Floor division for floats (rounding) 3/2 = 1 in v2 3/2 = 1.5 in v3 3//2 = 1 in both versions ** Exponentiation x%y modulo x|y bitwise (binary) OR of two integers x&y bitwise (binary) AND of two integers = Assignment == Test equality != Not equal (<> deprecated version) += -= *= /= //= or Logical and and Logical or not Logical not ~ Complement/negation applied to np.array bool lists Operations on different types: Result inherits general type Note: no ++, --, etc. iterators --------------------------------------------------------- Tuples (a, b, ...) Immutable Numbering from 0 (1,) Single tuple. (1) is just number 1 Function returns: a, b, c = ... return a, b, c --------------------------------------------------------- Lists [a, b, ...] [[1, 2], [4, 5]] List of lists = matrix Mutable Lists don't behave like vectors! List of lists don't behave like matrices! x[0][1] Matrix indices (not x[0,1] unless numpy list) x[i] = 3 Override [] Empty list []*N Empty list of size N l=l*2 Repetition (growing same entry) l*=2 l=l+[elem] Append/concatenate l+=[elem] l.extend('elem') l.append(elem) del l[i] Delete element l.index(elem) Where element elem in list Boolean presence if not a: Check empty list zip(l1,l2) Make pairs b=a Linked assignments. Changing one changes the other copy() Real copy of list *list List unpacking. --------------------------------------------------------- Deque (double-ended queues) Queues that are more efficient than lists. from collections import deque queue = deque([1,2,3]) queue.append(4) Append right queue.appendleft(4) Append left queue.pop() Delete right queue.popleft() Delete left --------------------------------------------------------- Strings "..." or '..' Immutable s[i] ith element len(s) Length str() Convert to string + Concatenate s=str(i).zfill(3) Padding s.upper() To uppercase s.lower() To lowercase .strip().split Split words r'\blabla' Raw string: \ treated as literal, not as escape u'blabla' Unicode string --------------------------------------------------------- Dictionaries {a, b, ...} a, b, ... String keys 'string':value String keys d['a'] Element by name d['c']=n Add element del x['a'] Delete key 'a' in d Check entry in d.keys() Extract keys as iterable d.values() Extract values as iterable d.items() Extract both as iterable sorted(d) Sorted by keys Also use: keys=(lambda x: ...) **dict Dictionary unpacking: gives values If assigned, only linked. For copy, use copy() Examples: for x in dict: print(x) for x in dict.keys(): for x in dict.values(): for x in dict.items(): % For both for substance in sorted(d): print(substance, d[substance]) --------------------------------------------------------- Sets S={a, b, c, ...} Dictionary without values. Immoral dictionary. Not ordered. Useful to delete repetitions. S={} Empty set s.add() Append value set(l) Transform to set Can also use s={*list} set(a)-set(b) Set difference intersection(a,b) union(a,b) difference(a,b) --------------------------------------------------------- Slicing x[start:end:step] end not included x[0] First x[-1] Last x[:2] Same as x[0:2] x[:] All elements. Same as x[:::] or x[::1] x[::2] All in steps of 2 x[::-1] Reversed (from start to end in reverse) x[2:] After 2, 2 included x[2:-1] From two, last excluded (!) slice(a, b, step) Named slice slice(5, None) [5:] x[named_slice] --------------------------------------------------------- Boolean masks (numpy arrays or pandas dataframes) Selection with mask: mask = x > 2 x[mask] or directly: x[x>2] For matrices: row_mask = M[:,0] > 2 col_mask = M[0,:] < 3 M[row_mask,:][:,col_mask] --------------------------------------------------------- List unpacking *a = list From a in list *a, b = list From a to last b a, *b = list First a, rest from b a, *b, c = list First a, rest, last c --------------------------------------------------------- Ranges range(n) List from 0 to n-1 range(a,b) List from a to b-1 range(a,b,d) List from a to b in step d; b not included r.append(n) Append element n r.pop() Remove last element r.pop(n) Remove n-1 element r.insert(n,'...') r.reverse() r=[] Clear --------------------------------------------------------- Iterator map(f,r) Elementwise f on list r v2 list(map(f,r)) Change iterator to list v3 Can apply functions directly on lists or arrays to produce arrays --------------------------------------------------------- Truth values False, 0, 0L, 0.0, empty strings, lists, etc. True 1, and \neq 0 number and non-empty strings, lists, etc. x is x x is not x elem in xlist elem not in xlist Comparisons: <, >, <-, >= == != (<> deprecated) Booleans: and, or, not ~ Take complements of np.array of True, False is Check if same object (address), not same value Example: y=0 if y: x/y else: print('Division not defined') --------------------------------------------------------- Conditional statement if [condition]: ... if [cond1]: ... elif [cond2]: ... elif [cond3]: ... else: ... Conditional assignment: exp = ... if ... else ... Chain conditions accepted: if 0 < x < 2: Example: print('horse' if x>=10 else 'duck' if 11=18 else False --------------------------------------------------------- While while [condition]: ... To be added: continue, break, pass Example: Fibonacci numbers f1, f2 = 1, 1 while f1<= 1000: f1, f2 = f2, f1+f2 --------------------------------------------------------- For loops for [item] in [sequence/iterable/container]: ... Mute iterator: for _ in ...: Example: for i in range(n): for key, val in d.item(): for x in list: Notes: - n not included in range (from 0 to n-1) - Ask yourself if loop can be re-written as list comprehension - Loop over container rather than indices --------------------------------------------------------- List comprehensions Don't loop on indices if you don't have to! Iterators: [expression for i in range(n)] [expression for i in list] [expression for i in list if condition] Same as for i in list: if condition: expression Examples: [i*i for i in range(5)] [i for i in list if boolean else ...] [x for x in X if P(x)] [f(x) for x in X if P(f(x))] [x**3 for x in range(n) if prime(x**3)] Pythonic filter(prime, [x**3 for x in range(n)] Non-pythonic print(list(filter(is_odd, aList))) Semi pythonic print([x for x in aList if x%2 == 1]) Pythonic List comprehensions can be nested. Generators: x=(expression for i in list) Not evaluated until prompted with x.next() Same behavior as 'yield' in loop --------------------------------------------------------- Functions def name(inputs): """ Name of function Description """ body return ... Default values: def name(x=2): One liner: def f(x): return 2*x yield Same as return for generators Notes: - Can pass function heads as input/argument to functions. - Multiple outputs: return a, b - Functions can be recursive - Inputs always passed as references - This implies that local changes to objects can be reflected globally. Very dangerous. Main program: def main(): Function stub: def name(...): pass No output: return None fcthead2=fcthead Renaming _, a, _ = fct() Dummy assignment Value memorization: from functools import cache @cache def myfunc(): --------------------------------------------------------- Lambda functions name=lambda vars: operation name(vars) Examples: mult=lambda x, y: x*y f = lambda x : [z + 5 for z in x] f = lambda x, y : [w + i for w, i in zip(x, y)] filter((lambda x: x>0), list) or directly on list --------------------------------------------------------- with command with object as acrn: acrn.... Same as acrn = object Useful for defining models in PyMC --------------------------------------------------------- Creating packages Save all functions in mypackage.py from mypackage import [fct] --------------------------------------------------------- Built-in functions type() Return object type isinstance(object,type) print() Or just put object variable and press shift-enter sum(list) int() complex() float() str() min(list) max(list) len(list) map(f,list) list(iterator) append() Method pop() Method range() type() complex(a,b) a+bj tuple(list) Tuple to list enumerate(list) For loops: give index, value in list for i, val in enumerate(list) filter(cond,list) zip(l1,l2) reduce() set() Transform to set ravel() Same as np.flatten pass Null operation --------------------------------------------------------- Plots with pylab from pylab import plot, show plot(list) plot(x,y) show() Show terminal figure Example: Plotting a function x=linspace(a,b,n) y=sin(x) plot(x,y) Options: ylim(a,b) xlim(a,b) xlabel(str) ylabel(str) Scatter plot: plot(x,y) or scatter(x,y) Labels: o Circles x Crosses ^ Triangles s Squares D Diamond - Line -- Dashed _ Dashed line : Dotted line -. Dash dot Colors: r Red g Green b Blue c Cyan m Magenta y Yellow k Purple w White --------------------------------------------------------- Plots with matplotlib import matplotlib.pyplot as plt plot(x) plot(x,y) show() title(str) xlabel(str) ylabel(str) xlim(a,b) ylim(a,b) axhline(Y_POS, ...) Add horizontal line axvline(X_POS, ...) Add vertical line axhspan(Y_START, Y_END, ...) Add horizontal shading axvspan(X_START, X_END, ...) Add vertical shading legend(['a','b'],loc='upper right') or label=str in plot figure() Re-initialize figure subplot(nmx) As in Matlab errorbar(x,y,xerr=...,yerr=...,fmt='...') Bar plots: plt.bar(bin_boundaries, bin_heights, width=) f = plt.figure() f.savefig("name.pdf", bbox_inches='tight') Ref: https://matplotlib.org Styling: style.use('named_style') https://matplotlib.org/3.1.0/gallery/style_sheets/style_sheets_reference.html https://python-graph-gallery.com/python-colors/ --------------------------------------------------------- Histograms Plots: plt.hist(data,binspec) plt.hist(data,bins=binspec) Bin specification: arange(a,b,d) Only bincounts Normalize as pdf: normed=1 Histogram counts: binheight, binboundary = np.histogram(data, bins=nbin) Options: bins=number or 'auto' range=(a,b) normed=True or False but deprecated density=True or False for pdf normalization --------------------------------------------------------- In and out a, b, ... = loadtxt('file.txt', unpack=True) From numpy Unpack: return columns in separate arrays Function ignores lines with # Assumes columns separated by tabs Assign raw string from keyboard: name = raw_input("Text...") Assign numerical value from keyboard: y = input("...") print "text" (v2) print a, b, c (v2) print(...) (v3) Space added automatically print(x,z,sep="...") for separator print(x,z,sep="") for no separator Formatting: "Text: %d" % x "Text: %d ... %2.2f ... %d" % (a, b, c) %d integer %xd integer with x width %x.yf floating with x integer and y decimals %x.ye scientific e notation %+... forces sign specification \n newline \t tab \# # \' \" FileHandle = open('File', mode) r Read w Write a Append FileHandle.read() Read all FileHandle.readline() Read line by line FileHandle.readlines() Read/parse all lines FileHandle.close() Close file FileHandle.write(string) numpy.loadtxt('file', delimiter=',') --------------------------------------------------------- Classes and objects Class with fixed attributes: class ClassName: var1 = ... var2 = ... def function(self, args): self. ... return ... Example: myvar = ClassName() myvar.var1 myvar.function(args) Notes: - self is the assigned var itself (or more precisely its memory address) - self = current instance of the class - self can be renamed to anything (and not a protected name) - Always there as extra arg but never called externally in list of args - Inside functions: self.var1, self.var2, ... - Functions inside classes are called methods - Attributes are public by default - Underscore attributes are private (need setters and getters for those) - However, Python has no real private attributes - they are still visible and usable Class with initialised attributes: class ClassName: def __init__(self, var1, var2): self.var1 = var1 self.age = var2 myvar = ClassName(a, b) Protected methods: __init__ Initialisation __call__ Built in method for calling class as a function __dict__ List of attributes @property decorator: See https://www.programiz.com/python-programming/property @dataclass decorator (New in 3.7): from dataclasses import dataclass @dataclass class DataClassCard: rank: str suit: str queen_of_hearts = DataClassCard('Q', 'Hearts') Notes: - No need for setter or getter methods - https://realpython.com/python-data-classes/ --------------------------------------------------------- Numpy pi e Functions are listable Array cannot be expanded after being declared (no appending) array(list) Array out of list array(list,dtype=type) Specify type int or float array(matrix) Matrix from list of lists matrix(' a b; c d ') Matlab notation astype Convert type x[a:b] a+1 to b or a to b-1 starting from 0 x[a:] a+1 to end x[:b] Beginning to b-1 x[:] or x All x[[1,2]] List of selected indices m[a,b] or m[a][b] m[a][b] more for Python lists m[2:4,3:6] m[2,3:6] m[2,:] or m[2] arange(a,b,s) Array range linspace(a,b,n) logspace(a,b,n) reshape(n,m) Reshape array in nxm matrix Method (comes after) zeros(n,type) Array of 0s. Type optional zeros([n,m],type) Matrix of 0s ones(n,type) Array of 1s ones([n,m],type) Matrix of 1s full((n,m), elem) Full matrix with element empty(n,type) Empty list empty([n,m],type) Empty matrix x.append(value) Append to x x.extend('hello') Append multiple elements fromfunction Build array from lambda function vectorize Make function listable x.size x.shape Dimensions x.shape = (1,2) Reshape x.reshape((a,b)) Reshape dims x.dim x.dtype Data type vsplit Split matrix vertically hsplit Split matrix horizontally flatten() Flattens Same as x.reshape(-1) Same as ravel np.clip where(array > 5) Find locations x.sort() Sort x Same as values=sort(values) key=len for sorting strings axis=0 or 1 for 2D arrays 2*a Listable a+b Array element add a+1 Listable add a*b Elementwise product dot(A,B) Dot product or matrix product cross(A,B) Cross product matmul(A,B) Matrix multiplication Can also use dot() Python 3.5: @ for matrix multiplication determinant(A) Determinant inv(A) Inverse inner Inner product solve Solve linear system sum(axis=0) Column sums sum(axis=1) Row sums cusum Cumulative sum (also with axis) transpose Transpose myarray.T Same myarray.H Conjugate transpose inv or .I det eig linalg.solve np.zeros_like np.ones_like array_equal(a,b) stack Combine matrices hstack Combine horizontally vstack Combine vertically Example: np.vstack((a,b)).astype(np.float32) Stack a on top of b (merge two sets of data/batches) median, mean, average std, var (normalized by N, not N-1) floor, ceil argmin, argmax intersect Notes: - numpy uses LINPACK - Explicit listable operation: b = array(map(sqrt,a),float) - A=B are linked, not copied. Use copy() for copy --------------------------------------------------------- Random numbers From numpy random.random() Uniform [0,1] random.random(size) Array with uniform RVs random.uniform(a,b) Random real in [a,b) random.choice(list) Random element from list random.randn(size) Normal array random.normal(mu,sigma) Gaussian random.randint(a,b) Random integer in [a,b] random.shuffle(list) Shuffle a list --------------------------------------------------------- Funcy package Functions for list manipulation merge join walk (like map) flip where remove concat select --------------------------------------------------------- String manipulation str=str.strip().split() Read from right to left str.str.join(a) + Concatenation len() Length of string zfill(n) Padding with n a[0:5] Slicing str(i) To string i --------------------------------------------------------- Curve fitting Linear least square: np.polyfit(x, y, degree) Fit polynomial np.polyval([m, c], xvec) Define polynomial Ref: https://mmas.github.io/least-squares-fitting-numpy-scipy Nonlinear least square: scipy.optimize.curve_fit popt, pcov = curve_fit(fct_fit, xdata, ydata, guess_params, dy_optional) --------------------------------------------------------- Standard data classes and methods .data All data (X) .data.shape Data size .feature_names Data labels .target Class labels (Y) .target.shape Label size .target_names Label names .astype(type) Type conversion --------------------------------------------------------- Pandas See Pandas notes. --------------------------------------------------------- Scipy stats import scipy.stats as stats poissonRV = stats.poisson poissonRV.pdf(x, scale = ...) poissonRV.pmf(x, ...) --------------------------------------------------------- Jupyter shortcuts Esc Enter command mode Esc m Text mode with latex enabled Esc y Code mode #, ##, ... Heading levels Esc B or b Insert cell below Esc A or a Insert cell above Esc D or d Delete cell - Bullet items 1. Enumerate items *text* Italics **text** Bold > text Quote $...$ Latex $$..$$ Displayed equation \\( .. )\\ Displayed equation %%latex More latex magic - can use anything like \[ \] Option + letter for greeks %run notebook.py Run external notebook %load file.py Load external script %%time Time cell %config InlineBackend.figure_format = 'retina' !ls Shell command jupyter nbconvert --to pdf jupyter nbconvert --to latex Insert image --------------------------------------------------------- Jupyter environments jupyter --paths jupyter kernelspec list jupyter kernelspec uninstall unwanted-kernel --------------------------------------------------------- Jupyter extensions pip3 show jupyter_contrib_nbextensions jupyter nbextension list System wide: jupyter contrib nbextension install User directory: jupyter contrib nbextension install --user --------------------------------------------------------- Style tips Space after commas, not before Space around operators (=, +, etc.), except with setting values in functions No space in slicing: a[1:3] PEP20: Readability counts References: https://www.python.org/dev/peps/pep-0008/ https://www.python.org/dev/peps/pep-0020/ The Zen of Python https://barry.warsaw.us/software/STYLEGUIDE.txt import this Some PEP8 conventions: _varname Distinguish private (non-public) names in code (eg in classes) fctname_ Distinguish from built-in function name __attribute Used in classes to relate attribute to specific class instance (name mangling) --------------------------------------------------------- Matlab engine in python (v2 not v3) import matlab.engine eng = matlab.engine.start_matlab() Start session (slow at first) eng.matlab_command Command use --------------------------------------------------------- Links General: - python.org - ipython.org/notebook.html Plotting: - matplotlib.org/gallery.html - python-graph-gallery.com Graphs/networks: - plot.ly/python/network-graphs/ - networkx.github.io