1. 1. python环境
  2. 2. 代码风格
  3. 3. 变量
    1. 3.1. 作用：
    2. 3.2. 赋值：
    3. 3.3. 引用
  4. 4. 数据类型
    1. 4.1. 标准数据类型：
    2. 4.2. 数据类型嵌套-对象
  5. 5. 运算符
    1. 5.1. 算数运算符
    2. 5.2. 比较运算符
    3. 5.3. 赋值运算符
    4. 5.4. 逻辑运算符
    5. 5.5. 成员运算符
    6. 5.6. 身份运算符
  6. 6. 程序结构
    1. 6.1. 分支结构
    2. 6.2. 循环结构
      1. 6.2.1. for循环
      2. 6.2.2. while循环
  7. 7. 函数
    1. 7.1. 参数
    2. 7.2. 返回值
    3. 7.3. 递归函数
    4. 7.4. 尾递归
    5. 7.5. 函数的使用
  8. 8. 字符串
    1. 8.1. 路径问题
    2. 8.2. 变量替换（字符串拼接、格式化）
    3. 8.3. 判断类函数
    4. 8.4. 内置函数
  9. 9. 类与对象
    1. 9.1. 类属性和方法
    2. 9.2. 装饰器
    3. 9.3. self
    4. 9.4. 继承
    5. 9.5. 构造函数
    6. 9.6. 封装
    7. 9.7. 多态
    8. 9.8. 类的函数转属性
    9. 9.9. 类的内置属性
    10. 9.10. 类的魔法函数
    11. 9.11. 抽象
    12. 9.12. 组装类
  10. 10. Package
    1. 10.1. module
    2. 10.2. the entrance of code
  11. 11. Error catch
  12. 12. Time mudule
  13. 13. OS module
  14. 14. Random module
  15. 15. Functional programming
    1. 15.1. lambda
    2. 15.2. high-level function
      1. 15.2.1. map
      2. 15.2.2. reduce
      3. 15.2.3. filter
      4. 15.2.4. sorted
      5. 15.2.5. Closure package
      6. 15.2.6. Decorator
      7. 15.2.7. Partial function
      8. 15.2.8. zip
      9. 15.2.9. enumerate
      10. 15.2.10. collections module
  16. 16. File operate
    1. 16.1. open
    2. 16.2. write
    3. 16.3. pickle module
  17. 17. Log
    1. 17.1. python logging module
    2. 17.2. usage:
      1. 17.2.1. specify default log level
      2. 17.2.2. use level function
      3. 17.2.3. use log function
    3. 17.3. by hand usage
  18. 18. Json
  19. 19. Multi threads
    1. 19.1. basic usage
    2. 19.2. daemon thread
    3. 19.3. shared variables
    4. 19.4. thread safe
    5. 19.5. deadlock
  20. 20. muiti processes
  21. 21. iterable/iterator
  22. 22. generator
  23. 23. coroutines
    1. 23.1. async/wait
    2. 23.2. concurrent.futures
      1. 23.2.1. ThreadPoolExecutor
      2. 23.2.2. ProcessPoolExecutor
  24. 24. regular expression
  25. 25. Xpath
  26. 26. net program
  27. 27. ftp
  28. 28. mail

python

python

之前学习过python，看书，视频等等，但是发现看的都不完整。所以这一次决定好好的来从头学习一遍。打算学好python，然后再学习go语言以及vue

python环境

  • python是一门解释型语言，也就是说不像c语言那样可以编译好了运行，python是边解释边执行的。所以源码可见，这个解释器就是python，我们在网上下载的python就是一个解释器。下载了并安装了python就拥有了一个python环境。

  • 查看python版本

    1
    python --version

  • 第一个函数print()
    print函数默认换行，如果不换行则需要加参数

    1
    2
    print("hello",end=" ")
    print("world")

代码风格

  • python代码的一个很明显的特点就是缩进。代码块是通过缩进来组织的，代码块使用冒号:来标识

  • 单行注释使用#

    1
    # 单行注释

  • 多行注释使用""" """

    1
    2
    3
    4
    """
    多行
    注释
    """

  • 一行代码拆分多行使用\

    1
    2
    3
    4
    5
    user = {"name":"snippet",\
            "age":"12",\
            "gender":"male"
            }
    print(user["age"])

  • 命名规范

    • 类命名：
      使用大驼峰的规范，即每个单词都以大写字母开头

      1
      2
      class UserInfo():
          name = "python"

    • 变量命名：

      1. 第一种：小驼峰，即第一个单词小写开头，其余字母大写开头
        1
        baiduUrl = "https://www.baidu.com"
      2. 第二种：所有字母全部小写，并用_连接，我比较喜欢这个
        1
        baidu_url = "https://www.baidu.com"

变量

作用：

  • 变量是一个字符串，用来保存数据的内存地址，指向内存中数据Data的位置。
  • 变量所保存的内存地址可以被改变，所以变量可变
  • 如果在程序中多处都要使用到这个数据，那么使用变量则可以只修改一个地方就行

赋值：

  • 由于python是解释型语言，所以在声明变量时，不需要指定变量的类型

    1
    name = "xiaotianquan"

  • 还可以一次性赋值多个

    1
    2
    name,age,like = "gou",2,"food"
    print(name,age,like)

引用

直接写变量名即可

数据类型

标准数据类型：

  1. 数字

    直接写数字，不加引号，加了引号作为字符串

    1
    age = 3

  2. 字符串 str

    用引号括起来的作为字符串，可以使用单引号和双引号
    这两种没有区别，因为没有变量置换（shell和perl中）这种操作，它使用的是拼接

  3. 布尔 True/False

    在python中，布尔值也可以作为数字使用
    True = 1, False = 0

    1
    2
    age = 3
    print(age+True)

  4. 列表 list

    用[]表示，可以存储不同类型的数据

    1
    2
    data = [1,2,3,4,5,6,7,"a","b"]
    print(data)

    切片操作

    • 使用:，取出指定范围的数据，可以使用函数id()查看取出数据的内存地址

      1
      list[start_index:end_index:step]
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      data1 = [1,2,3,4]

      # 作用为复制列表，使用频率高，有效的避免对原数组数据造成破坏
      data2 = data1[:]
      # 内存地址不同
      print(id(data1),id(data2))

      # 与data[:]作用相同，-1为最后一个元素的索引
      data3 = data1[0:-1]

      # 指定位置,前闭后开[)
      data4 = data1[0:2]

      # 指定步幅
      data4 = data1[::2]

      # 逆序
      data8 = data1[-1::-1]

      # 负数索引
      data5 = data1[-3:-1]

      其它

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16
      17
      18
      19
      20
      21
      22
      23
      24
      25
      26
      27
      28
      29
      30
      31
      32
      33
      34
      35
      36
      # 删除指定元素
      data6 = data1.remove(2)

      # 从后往前取出数据（删除）
      data7 = data1.pop()

      # 乘
      data9 = data1 * 2

      # 列表生成式
      data10 = [ i for i in range(1,10) ]

      # 嵌套
      data11 = [data,data10]

      # 循环取出嵌套数据
      for i in data11:
          for j in i:
              print(j)
          
      # 如果嵌套的两个list元素个数一致，可以使用一层嵌套即可取出所有值
      la = [1,2,3,4]
      lb = ["a","b","c","d"]
      lc = [la,lb]
      for i,j,k,l in lc:
          print(i,j,k,l)

      # 查看list元素个数
      len(data)

      # 全部为数字时，可以取最大值，最小值
      max(la)
      min(la)

      # 查看某个元素存在多少个
      la.count(1)

  5. 元组 tuple

    使用()，为一个不可修改列表，大部分用法与list一致

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    data = (1,2,3,4,"a","b")

    # 创建一个只有一个元素的tuple，后面需要一个逗号
    data = (1,)
    print(type(data))

    # 可以不加括号
    data = 1,
    print(type(data))
    data = 1,2,3,4
    print(type(data))

    # 创建空的tuple
    data = tuple()
    data = ()

    # 通过list创建tuple
    mylist = [i for i in range(1,9)]
    data = tuple(mylist)

    # tuple的特殊用法，两个元素交换
    a = 1
    b = 2
    a,b = b,a

  6. 集合 set

    • 数据有序，而且不可重复,具有取出重复的作用

    • 通过list和tuple来生成集合

      1
      2
      3
      4
      a = [1,2,3,4,2,1,3]
      b = set(a)
      print(type(b))
      print(b)

    • 通过{}来定义集合

      1
      2
      a = {1,2,2,5,3,4}
      print(type(a))

    • 集合没有索引，只能for循环取出元素

      1
      2
      3
      a = {i**2 for i in range(1,10)}
      for i in a:
          print(i)

    • 集合运算

      1
      2
      3
      4
      5
      6
      7
      8
      a = {i for i in range(1,5)}
      b = {i for i in range(3,7)}
      # 交集
      c = a.intersection(b)
      # 并集
      d = a.union(b)
      # 差集
      e = a.difference(b)

    • 让集合不可修改

      1
      2
      a = { i for i in range(1,5)}
      b = frozenset(a)

      其它 

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      # 生成奇数、偶数集合
      a = {i for i in range(1,10) if i % 2 == 0}
      b = {i for i in range(1,10) if i % 2 != 0}

      # 删除集合中的数，如果不存在则会报错
      sa = {i for i in range(1,10)}
      sa.remove(1)

      # 删除集合中的数，如果不存在不会报错
      sa.discard(11)

      # 集合中也有pop函数，但是是随机的
      sa.pop()
  1. 字典 dict

数据类型嵌套-对象

在list，tuple，set，dict中均可以存储对象

1
2
3
4
5
6
7
8
9
10
11
12
13
14
class User():
    name = "ls"
    age = 21
user = User()

a = [user]
b = (user,)
c = {user}
d = {"obj":user}

print(type(a),type(b),type(c),type(d))
print(a[0].name,b[0].name,d["obj"].name)
for i in c:
    print(i.name,i.age)

运算符

  • 算数运算符
  • 关系运算符
  • 赋值运算符
  • 逻辑运算符
  • 位运算
  • 成员运算符
  • 身份运算符

算数运算符

  • python没有自增自减运算符
  • 包括：
    • 加 +
    • 减 -
    • 乘 *
    • 除
      • 普通除法 /
      • 取余 %
      • 取整 //
  • 所有运算符都可以使用运算赋值的缩写：
    +=，-=，*=,/=,%=,//= 
    1
    2
    3
    a = 1
    a += 1
    print(a)
    已经有运算赋值这两个操作了，所以不能再赋值

比较运算符

比较运算的结果一定为一个布尔值：True/False

  • 等于 ==
  • 不等于 !=
    1
    print(3!=4)
  • 其它
    • >
    • >=
    • <
    • <=

赋值运算符

使用等号来赋值=，将数据data赋值给变量

逻辑运算符

返回结果为True/False

  • and
  • or
  • not
1
2
3
4
5
a = 1
if a == 3 or a == 2 :
    print("true")
else:
    print("false")

成员运算符

用来检测一个值或者变量是否在一个集合中,返回结果为True/False

  • in
  • not in
    1
    2
    3
    4
    5
    a = 1
    b = [i for i in range(1,5)]
    c = a in b
    d = a not in b
    print(c,d)

身份运算符

用来判断两个变量是否一致,返回结果为True/False

  • is
  • is not
    1
    2
    3
    4
    5
    6
    a = 1
    b = 1
    c = 2
    print(a is b)
    print(a is c)
    print(a is not c)

程序结构

  • 顺序
    程序从上到下依次执行
  • 循环
    for
    while
  • 分支
    if

分支结构

这里的条件表达式只要是返回结果为True/False即可

1
2
3
4
5
6
if 条件表达式:
    语句1
elif 条件表达式:
    语句2
else:
    语句3

举例1：

1
2
3
4
5
6
7
user = input("type your name: ")
if user == "root":
    print("you are the admin")
elif user == "narcissus":
    print("you are the normal user")
else:
    print("you are the user not in group sudo")

举例2：空字符串为假

1
2
3
4
if "":
    print("true")
else:
    print("false")

循环结构

for循环

用来遍历序列中的所有元素

1
2
for 变量 in 序列:
    语句

举例：

1
2
for i in range(1,10):
    print(i)
  • 循环中的条件跳转
    • break 终止循环
    • continue 跳过continue后的语句
    • pass 没有实际作用，用来占位

举例：

1
2
3
4
5
6
7
8
9
for i in range(2,20):
    if i % 2 == 0:
        print(str(i) + "能被2整除")
        continue
    elif i % 3 == 0:
        print(str(i) + "能被3整除")
        pass
    else:
        break

while循环

当条件成立时，一直循环。这个条件必须要有终止，否则会形成死循环。

1
2
while 条件表达式:
    语句

举例1：
求a，b的最小公约数

1
2
3
4
5
6
a = 2
b = 5
c = 1
while c % a != 0 or c % b != 0:
    c += 1
    print(a,b,c)

举例2：

1
2
3
4
5
6
7
8
9
i = 1
while i < 3:
    i += 1
    try:
        num =  int(input("num"))
        a = 5 / num
        print(a)
    except Exception as e:
        print("输入非零整数")

函数

  • 命名
    一般采用:动词_名词的形式

  • 意义
    复用代码

  • 定义

    1
    2
    def 函数名():
        语句

  • 调用

    1
    函数名()

  • 举例：

    1
    2
    3
    4
    5
    import time
    def get_time():
        now = time.strftime("%Y%m%d-%H:%M:%S")
        print(now)
    get_time()

参数

  • 分类方式1：

    • 形参：在写函数时，括号中的那个参数

      1
      2
      def get_name(name):
          print(name)

    • 实参：在函数调用时括号中传入的参数

      1
      get_name("小舞")

  • 分类方式2：

    • 位置参数：在传参时必须一一对应

      1
      2
      3
      def normal(name,age,like):
          print("{} is {} years old and like {}".format(name,age,like))
      normal("小舞",20,"star")

    • 关键字参数：在传参时不必一一对应，而是使用参数名来设置参数

      1
      2
      3
      def keyword(name,age,like):
          print("{} is {} years old and like {}".format(name,age,like))
      keyword(age=20, name="小舞", like="star")

    • 默认参数: 给参数设置默认值，如果不传则使用默认值，不过默认参数后面不能写位置参数

      1
      2
      3
      def default(a,b,c=1):
          print((a + b) / c)
      default(1,2)

    • 收集参数：在不确定会传入参数个数的情况下使用

      1. 如果后面还有位置参数，那么位置参数会作为关键字参数，而且位置不能变
      2. 如果有两个星号，那么表示收集所有参数包括关键字参数
        1
        2
        3
        4
        5
        6
        7
        8
        9
        10
        11
        12
        13
        14
        def collection(*count):
            sum = 0
            for i in count:
                sum += i
            return sum
        collection(1,2,3,4)


        def collection(*count,another=1):
        sum = 0
        for i in count:
            sum += i
        return sum + another
        collection(1,2,3,4,another=5)

    • 分解参数：

      1. 将list和元组作为参数传给函数时

        1
        2
        3
        4
        5
        6
        7
        8
        9
        def expend(*args):
            sum = 0 
            for i in args:
                sum += i
            return sum
        a = (i for i in range(1,10))
        b = [i for i in range(1,10)]
        print(expend(*a))
        print(expend(*b))

      2. 将字典作为参数传给函数时

        1
        2
        3
        4
        5
        6
        7
        def expend(**args):
            sum = 0
            for i in args:
                sum += args[i]
            return sum
        c = {"a":1,"b":2,"c":3}
        expend(**c)

返回值

  • 函数可以有返回值，也可以没有返回值，使用return来返回执行结果
  • 但是在python中还是推荐有返回值，如果没有实际值返回可以返回一个None
  • 函数一旦执行return之后函数就会结束，return后面的语句不会执行

举例：

1
2
3
4
5
6
7
8
9
10
11
12
13
def sum(a,b):
    return a + b
a = sum(1,2)
print(a)

c = 1
def plus():
    global c
    c += 1
    return None
# 函数没有返回值，每次调用函数c的值就自增1
plus()
print(c)

计算九九乘法表：我写的挺复杂

1
2
3
4
5
6
7
8
for y in range(1,10):
    x = 1
    while x <= y:
        print(str(x) + "*" + str(y) + "=" + str(x*y) \
        + " ",end='')
        x += 1
        if x > y:
            print(end="\n")

简单写法

1
2
3
4
for y in range(1,10):
    for x in range(1,y+1):
        print('{}*{}={}\t'.format(x,y,x*y),end="")
    print()

三角形

1
2
3
4
5
6
for y in range(1,10):
    for x in range(1,10-y):
        print("   ",end="")
    for x in range(1,2*y):
        print(" 1 ",end="")
    print()

递归函数

函数直接或者间接调用自己，需要有结束条件

优缺点：

  • 优点：书写理解简单
  • 缺点：执行速度慢

写法：

  • 结束条件

    1
    2
    3
    4
    5
    6
    7
    def plus(num):
        if num == 1:
            return 1
    # 这个函数调用时，传入1，得到plus(1) = 1
    # 不能这么写，这么写是错误的
    def plus(num):
        plus(1) = 1

  • 返回自己

    1
    2
    3
    4
    5
    def plus(num):
        if num == 1:
            return 1
        return num + plus(num-1)
    plus(100)

  • 举例：斐波拉契数列

普通写法

1
2
3
4
5
6
7
8
9
10
# 1, 1, 2, 3, 5, 8, 13.....
def fun(num):
    a = b = i = 1
    sum = 1 if num == 1 else 2
    while i < num-1:
        a,b = b,a+b
        i += 1
        sum = sum + b 
    print("第{}个斐波拉契数为：{}，总数为：{}".format(num,b,sum)) 
fun(3)

使用递归写法

1
2
3
4
5
6
7
8
9
10
# 1, 1, 2, 3, 5, 8, 13.....
def fib(num):
    if num == 1 or num == 2:
        return 1
    return fib(num-1) + fib(num-2)

def fib_list(num):
    for i in range(1,num+1):
        print(fib(i))
fib_list(10)

尾递归

对比普通写法和递归的写法，递归函数在调用40次时已经停止调用了，而且在求同一个斐波拉契数的时候，普通写法速度秒杀递归写法。普通写法单线程求第十万个斐波拉契数不到一秒，第一百万个斐波拉契数约六秒。所以递归函数不适合复杂计算

递归函数为什么慢？因为递归函数运行时，每一次调用都会返回一个表达式，不能返回一个确定的值，那么原函数的参数，逻辑结构等这些都不能修改，每次计算机在调用函数时会使用堆栈，每调用一个函数会增加一层栈帧，所以当递归过程多次调用函数的时候可能会导致大小有限的堆栈溢出。优化方式之一是尾递归

为什么尾递归可以优化？尾递归与递归都需要调用自身，看起来没多大区别（我刚开始也没看懂），但是尾递归每次都不是返回的表达式，而是返回的一个函数，至于计算结果，那是通过变量的形式进行传递的，这么做的好处就是不用开新的栈，而是刷新原函数，将参数传递进去就好了。所以做到了优化，但是python中即使是使用了尾递归，次数仍然只能达到999次。并不是真正意义上的尾递归

在很多时候我们都是在刷新变量的值，而并不是重新用一个别的变量

1
2
3
4
sum = 0
for i in range(1,10):
    sum += i
print(sum)

尾递归的斐波拉契函数写法：

1
2
3
4
5
6
# 1, 1, 2, 3, 5, 8, 13.....
def fib(num, first=1, second=1, sum=2):
    if num == 2 or num == 1:
        return second,sum
    return fib(num-1, second, first+second, sum+first+second)
fib(7)

利用三目运算简写

1
2
3
def fib(num, first=1, second=1, sum=2):
    return 1 if num == 1 else(sum  if num == 2 else fib(num-1, second, first+second, sum+first+second))
fib(7)

通过上面尾递归的写法，可以看出函数调用自身的作用类似于计数，真正计算的过程在参数中。其实类似于普通函数的写法

函数的使用

在python中一切皆是对象，所以函数也是对象，函数名指向的是代码段的内存地址

  • 作为元素使用：放在list，tuple，set，dict中

    1
    2
    3
    4
    def user():
        print("hello")
    one = user()
    a = [one]

  • 作为参数传递给函数

  • 可以被赋值

  • 作为函数的返回值

字符串

在linux相关的运维上，有很多都是处理字符串，所以字符串有点重要的

路径问题

在很多情况下，输入路径需要各种转义才行，为了避免这种麻烦，使用r

1
2
location = r'c:\user\kite\desktop'
print(location)

变量替换（字符串拼接、格式化）

  • %

    1
    print("i like %s and %s"%("cat","dog"))

  • format函数，使用{}

    • 直接使用

      1
      print("i like {} and {}".format("cat","dog"))

    • 使用变量

      1
      2
      a = "i like {} and {}"
      print(a.format("cat","dog"))

    • 指定位置

      1
      2
      a = "i like {0} and {0} and {1}"
      print(a.format("cat","dog"))

    • 使用命名：

      • 我觉得这种还不错，虽然字多了一点，但是很清晰
        1
        2
        a = "i like {animal} and {animal} and {drink}"
        print(a.format(animal="cat",drink="juice"))
      • 在传入的参数比较多时，使用字典是最好的,需要使用到参数分解，这个是目前我觉得最好的
        1
        2
        3
        a = "i like {animal} and {animal} and {drink}"
        b = {"animal":"cat","drink":"juice"}
        print(a.format(**b))

    • 指定数字长度

      1
      2
      a = "pi is {:.2f}"
      print(a.format(1.1314))

判断类函数

函数以is开头，返回值为True/False

内置函数

字符串的关键字为str，通过str来查看帮助文档

1
help(str)

举例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
url = r"https://www.baidu.com"
# 查找，索引，未找到返回-1
url.find("http")
# 从指定位置开始查找，优化查找速度
url.find("d",10)

# 从右边查找，查找字符在结尾时优化查找速度
url.rfind("m")

# 查找，索引，未找到报错
url.index("h")

# 开头，结尾，返回True，False
url.startswith("https")
url.endswith("com")

# 是否全大小写,返回True，False
url.isupper()
url.islower()

# 大小写转换
url.upper()
url.lower()

# 首字母大写
url.capitalize()

# 不带参数代表空格，去除开头结尾空格
error = r"   https://www.baidu.com   "
a = error.strip()

# 带参数去除开头结尾指定字符
url.strip("https")

类与对象

类通过class来定义，对象由类实例化而来。类的作用在于为对象提供一个模板，它包含了属性和函数。最重要的是类提供了继承，封装，多态这三大特性。

1
2
3
4
5
6
# 定义类
class User():
    pass

# 实例化
xiaolu = User()

查看类的所有方法：__dict__

1
str.__dict__

类属性和方法

在类定义之后，就在内存中保存了类的属性和方法，可以直接调用

1
2
3
4
5
6
class User():
    name = "jounr"
    def get_name(self):
        return self.name
print(User.name)
print(User.get_name(User))

  • 实例方法：使用最多的方法，需要将类实例化成为对象，通过对象调用方法

  • 静态方法：

    • 不需要实例化，通过类直接调用
    • 静态方法包装在staticmethod类的对象中，定义中没有self，可直接通过类来调用方法
      1
      2
      3
      4
      5
      6
      class MyClass():
          def tellme():
              print("i am a static method,without self")
          tellme = staticmethod(tellme)

      MyClass.tellme()

  • 类方法：

    • 不需要实例化，通过类直接调用，如果调用需要传参的函数，则将类自己作为参数传递进去
    • 类方法包装在classmethod类的对象中，参数为cls，虽然可以通过对象来调用，但是实际上cls还是会关联到类的方法
      1
      2
      3
      4
      5
      6
      class MyClass():
          def tellme(cls):
              print("i am a class method,with args sls")
          tellme = classmethod(tellme)

      MyClass.tellme()

装饰器

  • 上面的类方法和静态方法，需要手动写，但是可以通过使用装饰器来自动包装
  • 用来替代手工包装，可用于包装任何可调用对象，还可用于方法和函数。装饰器可以有多个，指定多个时，应用的顺序与列出的顺序相反
    1
    2
    3
    4
    5
    class MyClass():
        @staticmethod
        def tellme():
            print("new decorator")
    MyClass.tellme()

self

  • self作为一个参数，常常用在类的函数中，但并不是所有类的函数都需要这个参数
  • self并不是一个关键字，也可以用其它的替代
  • 如果这个函数需要访问类的属性，那么需要self这个参数，它用来指代实例自身
  • 如果这个函数不需要访问类的属性，那么不需要self这个参数
  • 在有self的时候，首先会看函数自己是否定义属性，如果没有，则访问类的属性
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    class User():
        age = 20
        id = 1
        qq = 102832343
        def info():
            print("hello")
        def get_age(self):
            return self.age
        def get_qq(self):
            self.qq = 2312323123
            return self.qq,self.age
    one = User()
    one.info()
    print(one.get_age())
    print(one.get_qq())

继承

  • 继承父类的属性，函数等。避免了代码的冗余，也可重写父类的方法

  • 将父类作为参数传递给子类

  • 子类可以继承自多个类

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    class User():
        name = "one"
        age = 20
        def get_name(self):
            return self.name
    class Customer():
        id = 1
    class Realone(User,Customer):
        pass

    a = Realone()
    a.get_name()
    print(a.id)

  • 子类扩充父类的方法：使用super()调用父类方法

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    class User():
        name = "journal"
        def get_name(self):
            print(self.name)
    class SomeOne(User):
        def get_info(self):
            super().get_name()
            print("attend super")

    a = SomeOne()
    a.get_info())

  • 检测是否为子类

    1
    print(issubclass(Realone,User))

构造函数

  • 在类实例化的时候第一个自动调用的函数，作用是给对象初始化一些数据

  • 必须有参数self,函数名为__init__

    1
    2
    3
    4
    5
    6
    class User():
        def __init__(self):
            self.name = "lixingyun"

    a = User()
    print(a.name)

  • 通过继承，扩展父类的构造函数

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    class User():
        def __init__(self,name):
            self.name = name
            print(name)

    # 直接写父类的init方法，只能写一个父类，不推荐使用
    class One(User):
        def __init__(self,name):
            # 先调用父类的init函数
            User.__init__(self,name)
            print("initial finish")

    # 可以自动向上寻找父类的init方法，父类没有写死，推荐使用
    class Two(User):
        def __init__(self,name):
            super(Two,self).__init__(name)
            print("initial finish")

    a = One("小小")
    b = Two("小小")

  • 构造函数默认被继承

  • 如果定义了自己的构造函数，那么父类的构造函数不再被继承。也叫重写父类函数

封装

封装即为对对象成员进行访问限制

  • 私有：最高级别的限制，只能在当前类或者对象中访问。通过在变量前加__实现

    1
    2
    3
    4
    5
    6
    class User():
        __name = "喜洋洋"
        __age = 20
    one = User()
    # 无法直接访问
    print(one.name)

    其实私有也不是真的无法访问，还是可以访问的

    1
    print(one._User__age)

  • 受保护的：在外部不可访问，可以在内部和子类中访问。通过在变量前加_实现

  • 公有的：任何地方都可以使用

多态

  • 同一个对象在不同情况下有不同的状态
  • 在python中是一种设计思想，并没有语法要求，其实我也搞不懂

类的函数转属性

类中的get，set，del函数可以转换为属性

1
2
3
4
5
6
7
8
9
10
11
12
class User():
    def __init__(self,name):
        self.name = name
    def get_name(self):
        return self.name
    def set_name(self,name):
        self.name = name

one = User("小小")
one.get_name()
one.set_name("小青")
one.get_name()

可以使用property(get,set,del)函数对get，set，del封装。将函数转换为一个属性值

1
2
3
4
5
6
7
8
9
10
11
12
13
class User():
    def __init__(self,name):
        self.name = name
    def get_name(self):
        return self.name
    def set_name(self,name):
        self.name = name
    sname = property(get_name, set_name)

two = User("小二")
two.sname
two.sname = "可可"
two.sname

类的内置属性

  • __dict__： 显示类的属性和方法
  • __doc__： 类的说明文档
  • __name__： 类的名称
  • __bases__： 类的所有父类

类的魔法函数

不需要手动调用，达到触发条件，自动调用

  • __init__： 初始化类的属性

  • __call__： 当对象被当做函数使用的时候，调用此函数

    1
    2
    3
    4
    5
    class User():
        def __call__(self):
            print("此时对象被当做函数调用了")
    one = User()
    one()

  • __str__： 当对象被当做字符串使用的时候，调用此函数

    1
    2
    3
    4
    5
    class User():
        def __str__(self):
            return "此时对象被当做字符串使用了"
    one = User()
    print(one)

  • __getattr__： 当访问一个不存在的属性时，调用此函数

    1
    2
    3
    4
    5
    6
    class User():
        def __getattr__(self,method):
            return "此函数不存在"

    one = User()
    print(one.age)

抽象

  • 抽象方法

定义了方法，但没有实现

1
2
3
class User():
    def get_name(self):
        pass

  • 抽象类

    • 必须导入模块abc
    • 抽象类不可以实例化，必须继承后实例化，继承的子类需要实现所有的继承来的抽象方法
    • 作用为设定类的标准
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
import abc

class User(metaclass=abc.ABCMeta):

    # 定义抽象方法
    @abc.abstractmethod
    def get_name(self):
        pass

    # 定义类抽象方法
    @abc.abstractclassmethod
    def get_age(self):
        pass

    # 定义静态抽象方法
    @abc.abstractstaticmethod
    def get_id(self):
        pass

    # 定义普通方法
    def set_qq(self,qq):
        self.qq = qq

组装类

函数名可以作为变量使用

1
2
3
4
5
6
7
8
class User():
    pass
def he(self):
    print("hello")

User.he = he
one = User()
one.he()

Package

package is a collection of modules, which must be included with a file called __init__.py

  • how to import a package

    • import all include __init__.py

      1
      2
      3
      4
      import package_name

      package_name.function_name()
      package_name.class_name.function_name()

    • only import specify module,except __init__.py

      1
      2
      3
      4
      import package_name.module_name as md1

      md1.function_name()
      md1.class_name()

    • another way import should be mentiond

      1
      from package_name import *

      the way import only import function and class which in __init__.py

module

a module is a file contains python code.it’s suffix with .py

  • why we need module?

    • shrink program,make it easy to maintain
    • module can be reused
    • used as a namespace,reduce name conflict

  • how to use module?

    • import directily

      1
      2
      3
      4
      import module_name

      module_name.function_name()
      module_name.class_name()

    • if the module name prefix with num such as 2.py

      1
      2
      3
      4
      5
      6
      import importlib

      variable = importlib.import_module("02")

      variable.function_name()
      variable.class_name()

    • use as rename module when the name of module is long

      1
      import module_name as somename

    • import module’s class or function,not whole module

      1
      2
      3
      4
      from module_name import someclass,somefunction

      variable1 = someclass()
      variable2 = somefunction()

    • import all class and function

      the benifit of it is writing class or function directily without module name prefix

      1
      2
      3
      from module_name import *

      variable = someclass()

  • where modules store?

    • default location

      1
      2
      3
      4
      import sys

      for i in sys.path:
          print(i)

    • add own module location

      1
      2
      3
      import sys

      sys.path.append(directory)

the entrance of code

  • in genreal,all entrance of code should be started with this
  • it can control whether to execute some code,if used as a code which will be execute,if used as a module which will not be execute
1
2
if __name__ == "__main__":
    print("first execute code")

Error catch

some days ago,i really don’t know why we need to catch error.in my mind,i think we should not write code with error.until one day,i get a problem,i need to create a temporary table first and then delete it every time the script run.and the error appeard.some times the table isn’t exist,when execute the delete code,the program broken.at last i use error catch to avoid the error

eg:get user type in

1
2
3
4
5
6
try:
    num = int(input("num:"))
    result = 100 / num
    print(result)
except:
    print("not a none zero number")
  • syntax
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    try:
        ...
        # can raise error by hand
        raise some_error_type
    # if it is the error below(tuple),execute this code
    except (error_type1,error_type2...):
        ...
        # exit() can exit the program
        exit()
    # we can print error infomation
    except ZeroDivisionError as e:
        print(e)
    # Exception is the supper of all except type
    except Exception as e:
        ...
    # if not in the error above execute this code
    else:
        ...
    # what ever error happend,will execute this code
    finally:
        ...

Time mudule

1
2
3
4
5
6
7
8
9
10
11
12
13
import time

# get time now with format
now = time.strftime("%Y-%m-%d %H:%M:%S")

# another way to get the format time
hour = time.localtime().tm_hour

# get the timestamp
unique = time.time()

# sleep
time.sleep(second_number)
1
2
3
4
5
6
7
8
9
10
11
import timeit

# calcute program execute time with number times repeat
t1 = timeit.timeit(stmt="[i for i in range(10)]",number=100)

def number():
    lt = []
    for i in range(10):
        lt.append(i)

t2 = timeit.timeit(stmt=number,number=100)
1
2
3
4
5
6
7
8
9
10
11
12
13
import datetime

# convert string to date
a = datetime.date(2020,10,10)
print(a)
b= datetime.time(10,10,10)
print(b)
c = datetime.datetime(2020,10,10,10,10,10)
print(c)

# get the current datetime
d = datetime.date.today()
print(d)

OS module

associated with operate system

three important modules:

  • os
  • os.path, used for path
  • shutil, used for directory,file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import os

dir = os.getcwd()

os.chdir("../")

# get file and dir of current directory with out hidden file
lt = [i for i in os.listdir() if not i.startswith(".")]

# execute shell command
os.system("htop")

# get system environment variables
os.getenv("SHELL")
1
2
3
4
5
6
7
8
9
10
11
12
13
14
import os.path as op

absc = op.abspath(".")

# used for different operate system
d1 = "/home/narcissus"
d2 = "1.txt"
os.join(d1,d2)

# os.path.split return a tuple,so can be used like this
d1,d2 = op.split("/home/narcissus/1.txt")

# whether a path exists
op.exists("/home/narcissus/1.txt")
1
2
3
4
5
6
7
8
9
10
11
import shutil

shutil.copy("1.txt","2.txt")

shutil.move("1.txt","2.txt")

# archive all file of Desktop into test.tar
shutil.make_archive("/home/narcissus/Desktop/test","tar","/home/narcissus/Desktop/")

# unarchive file
shutil.unpack_archive("/home/narcissus/Desktop/test.tar",".","tar")

Random module

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
import random

# return a number in the interval [0,1)
r1 = random.random()

# return a number in the interval [0,100)
r2 = int(random.random()*100)

# random choice in the interval [1,100)
l1 = [i for i in range(0,100)]
random.choice(l1)

# shuffle list
l1 = [i for i in range(1,10)]
print(l1)
random.shuffle(l1)
print(l1)

# random integer
random.randint(1,100)

Functional programming

features:

  • function can be used as a parameter
  • function can be used as return value

lambda

lambda exprssion

  • begin with lambda
  • have zero or more parameter
  • use : split lambda with expression
  • no return keyword
1
2
3
4
5
la = lambda x: x * 100
la(10)

lb = lambda x,y,z: x + y*10 + z/2
lb(2,4,8)

high-level function

1
2
3
4
5
6
def get_name():
    print("小小")

# function is a variable
variable = get_name
variable()
1
2
3
4
5
6
7
8
9
10
11
12
13
# normal function
def plus(num):
    return num * 100
def add(num):
    return plus(num) + 100
add(3)

# high_level function
def plus(num):
    return num * 100
def high_level(fun,num):
    return fun(num) + 100
high_level(plus,2)

map

map(func,*iterables)

1
2
3
4
5
6
7
8
9
10
11
12
13
l1 = [i for i in range(10)]
print(l1)

# use lambda expression ,ten times a num
mult = lambda num: num * 10

# use map to generate a class map
l2 = map(mult,l1)
print(l2)

# class object can be iterabled
l3 = [i for i in l2]
print(l3)

reduce

notice:

  • need import module
  • function need two parameters
  • function need return value

principle:
reduce(1,2,3) = f(f(1,2),3)

1
2
3
4
5
6
7
8
from functools import reduce

l1 = [i for i in range(10)]

plus = lambda x,y: x + y

l2 = reduce(plus,l1)
print(l2)

filter

notice:

  • function need return a boolean
  • filter return a class object which can be iterabled

filter(function,*iterables)

1
2
3
4
5
6
7
8
l1 = [i for i in range(10)]

fit = lambda x: x % 2 == 0

l2 = filter(fit,l1)

l3 = [i for i in l2]
print(l3)

sorted

number sorted

1
2
3
4
5
6
7
8
9
10
11
import random

l1 = [i for i in range(10)]
random.shuffle(l1)
print(l1)

l2 = sorted(l1)
print(l2)

l3 = sorted(l1,reverse=True)
print(l3)

charactor sorted

1
2
3
4
5
6
7
8
9
10
l1 = ["sda","dev","HOME","Alian","alias"]

l2 = sorted(l1)
print(l2)

l3 = sorted(l1, key=str.lower)
print(l3)

l4 = sorted(l1, key=str.lower, reverse=True)
print(l4)

Closure package

B function defined inside A function,and B function as a return vlaue,all parameters of A can be used by B,it is called closure

1
2
3
4
5
6
7
8
9
10
11
def A(*args):
    def B():
        sum = 0
        for i in args:
            sum += i
        return sum
    return B

l1 = [i for i in range(20)]
one = A(*l1)
one()

Decorator

Decorator is a high-level function,i can return a function.

feature:

  • extend origion function without change its code
  • use @ to called a decorator
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import time

def now(func):
    def pf(*args,**kwargs):
        print("Time:",time.ctime())
        return func(*args,**kwargs)
    return pf

def line(func):
    def pf(*args,**kwargs):
        print("="*50)
        return func(*args,**kwargs)
    return pf

# use system decorator @
@now
@line
def he(name):
    print("hello",name)

he("小小")


# by hand execute without system decorator @
return_fun = now(he)
return_fun("小小")

Partial function

return a new function with some of its parameters fixed

1
2
3
4
5
6
7
8
9
10
11
import functools

# convert 16 to 10
int16 = functools.partial(int,base=16)
int16("f")


# we can realize this by hand
def int16(num,base=16):
    return int(num,base)
int16("f")

zip

combine two iterable object into several tuples

1
2
3
4
l1 = [i for i in range(1,10)]
l2 = [i for i in range(100,110)]
l3 = [i for i in zip(l1,l2)]
print(l3)

enumerate

give each iterable object a index and return several tuples,notice that the index can be customized

1
2
3
4
5
l1 = [i for i in range(1,10)]

l2 = [i for i in enumerate(l1,start=1)]

print(l2)

collections module

  • namedtuple: it can limit the number of tuple
  • deque: it is much faster than list
  • defaultdict: no error report when query none exist key
  • Counter: regard string as a iterable dict
1
2
3
4
5
6
import collections

circle = collections.namedtuple("circle",["x","y","r"])
p = circle(3,4,5)
if p.x**2 + p.y**2 == p.r**2:
    print("it is a circle")
1
2
3
4
5
import collections

ql = collections.deque()
ql.append(1)
print(ql)
1
2
3
4
5
6
7
import collections

func = lambda : "no key exist"
dd = collections.defaultdict(func)
dd["one"] = "first"
dd["one"]
dd["four"]
1
2
3
4
5
6
import collections

cr = "ksjflajowielzxkjowieof1242lkjlk"
cd = collections.Counter(cr)
print(cd)
print(cd["j"])

File operate

open

  • mode
    • r: read only
    • w: new write,will erase origional file
    • x: open exist file,if not error reportd
    • a: append mode
    • b: binary write
    • t: file mode open
    • +: read and write

in this way, we need to close file handle by hand

1
2
3
4
5
dir = r"/home/narcissus/Desktop/遍历游标的三种方式.md"
f = open(dir,"r")
for i in f:
    print(i)
f.close()

in this way, file handle will be closed automatically

1
2
3
4
dir = r"/home/narcissus/Desktop/遍历游标的三种方式.md"
with open(dir,"r") as f:
    for i in f:
        print(i)

read only one line,so should be carefully

1
2
3
4
5
6
dir = r"/home/narcissus/Desktop/遍历游标的三种方式.md"
with open(dir,"r") as f:
    i = f.readline()
    while i:
        print(i)
        i = f.readline()

read all lines

1
2
3
4
5
6
7
8
9
10
11
dir = r"/home/narcissus/Desktop/遍历游标的三种方式.md"
with open(dir,"r") as f:
    i = f.readlines()
    for j in i:
        print(j)

# or use this
with open(dir,"r") as f:
    i = list(f)
    for j in i:
        print(j)

read each word

1
2
3
4
5
6
dir = r"/home/narcissus/Desktop/遍历游标的三种方式.md"
with open(dir,"r") as f:
    word = f.read(1)
    while word:
        print(word,end="")
        word = f.read(1)

write

  • write: only support string
  • writeline: can write list directily
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import os

dir = r"/home/narcissus/Desktop/1.txt"
txt = os.listdir("/home/narcissus")
needed = [i for i in txt if not i.startswith(".")]
with open(dir, "w") as f:
    # write whole list once without space
    f.writelines(needed)

    # can not write list directily,but can use loop to write new line per element
    # for i in needed:
    #     f.write(i)
    #     f.write("\n")
with open(dir,"r") as f:
    info = f.readlines()
    print(info)

pickle module

it is used for persistence file, can save program’s information to disk while the program runing

feature:

  • it can store infomation immediately
  • it can store infomation with format unchanged

function:

  • pickle.dump: write
  • pickle.load: read

notice:
use binary mode to write or read file

1
2
3
4
5
6
7
8
9
10
import pickle,os

dir = r"/home/narcissus/Desktop/1.txt"
txt = os.listdir("/home/narcissus")
info = [i for i in txt if not i.startswith(".")]
with open(dir,"wb") as f:
    pickle.dump(info,f)
with open(dir,"rb") as f:
    get = pickle.load(f)
    print(get)

Log

why we need log?
recording some information at important points, suitable log information can let us to know program running statius
but that isn’t to say log is much more much better,because io operation will slow down program.

there are several log level:

  • debug
  • info
  • notice
  • warning
  • error
  • critical
  • alert
  • emergency

log include:
[time location level content]

good log module:

  • log4j -> java
  • log4php -> php
  • logging -> python

python logging module

level:

  • debug
  • info
  • warning
  • error
  • critical

default log level:

  • specify a log level in program
  • only log level which equal or high above than specified level will be write in log

usage:

specify default log level

only logging once at first time executes

feature:

  • specify the default log level
  • output: sys.stderr
  • default level: warning
  • format: level:log_name:content
1
logging.basicConfig(**kwargs)

eg:

1
2
3
4
5
import logging

dir = r"/home/narcissus/Desktop/info.log"
log_format = "%(asctime)s   %(levelname)s   %(message)s"
logging.basicConfig(filename=dir, level=logging.DEBUG, format=log_format)

use level function

1
2
3
4
5
logging.debug(msg, *args, **kwargs)
logging.info(msg, *args, **kwargs)
logging.warning(msg, *args, **kwargs)
logging.error(msg, *args, **kwargs)
logging.critical(msg, *args, **kwargs)

eg:

1
2
3
4
import logging

logging.debug("this is a debug log")
logging.info("this is a info log)

use log function

1
logging.log(level, *args, **kwargs)

eg:

1
2
3
4
import logging

logging.log(logging.DEBUG,"this is a debug log")
logging.log(logging.INFO,"this is a info log")

complete example

1
2
3
4
5
6
7
8
9
10
11
12
import logging

dir = r"/home/narcissus/Desktop/info.log"
log_format = "%(asctime)s\t%(levelname)s\t%(lineno)d %(message)s"
logging.basicConfig(filename=dir, level=logging.DEBUG, format=log_format)


logging.debug("this is a debug log")
logging.info("this is a info log")

logging.log(logging.DEBUG,"this is a debug log")
logging.log(logging.INFO,"this is a info log")

by hand usage

  • logger: a interface of log produces log
  • handler: decide where log stores
  • filter: what should be displayed
  • formatter: format

logger:

1
2
3
4
5
6
7
8
9
Logger.setLevel()
Logger.addHandler()/Logge.removeHandler()
Logger.addFilter()/Logger.removeFilter()
Logger.debug()/info/error
Logger.exception()
Logger.log()


logging.getLogger()

and so on …..

Json

convert:

  • json.dumps(): python -> json
  • json.loads(): json -> json

read and write with file:

  • json.dump(): write to file
  • json.load(): read from file
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
import json,os

data = {
    "user": "小小",
    "age": 20,
    "like": ["juice","apple","cherry"],
    "score": {
        "chinese": 80,
        "math": 90,
        "english": 100
    }
}

jd = json.dumps(data)
print(type(jd),jd)
nd = json.loads(jd)
print(type(nd),nd)

dir = r"/home/narcissus/Desktop"
os.chdir(dir)
with open("test.js","w") as f:
    json.dump(data,f)

with open("test.js","r") as f:
    file = json.load(f)
    print(file["score"]["chinese"])

Multi threads

process vs thread:
process totally different,but thread will share something.
process builds with threads

basic usage

syntax:

1
2
3
4
5
6
7
8
9
import threading
# args is tuple
td = threading.thread(target=function_name,args=())
# set thread name,it is optional
td.setName("thread1")
# start thread
td.start()
# waiting for thread finish
td.join()

notice:
the type of args is tuple

daemon thread

when process stops the daemon thread will stop immediately

syntax:

1
2
3
4
5
6
7
import threading
# args is tuple
td = threading.thread(target=function_name,args=())
td.daemon = True # or td.setDaemon(True)
td.start()
# waiting for thread finish
td.join()

shared variables

it is appeard when several threads access one variable at the same time, it will lead wrong result. In order to resolve this problem, we can lock the use of variable

syntax:

1
2
3
4
lock = threading.Lock()
lock.acquire()
lockd_variable
lock.release()

without lock

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import threading

sum = 0
num = 1000000

def add(num):
    global sum
    for i in range(num):
        sum += 1
    return sum

def miu(num):
    global sum
    for i in range(num):
        sum -= 1
    return sum
# normal usage will return sum = 0
# miu(num)
# add(num)
# print(sum)


# use thread access one variable at the same time will lead wrong result
def calcu():
    td1 = threading.Thread(target=add,args=(num,))
    td2 = threading.Thread(target=miu,args=(num,))

    td1.start()
    td2.start()
    td1.join()
    td2.join()

if __name__ == "__main__":
    calcu()
    print(sum)

use lock

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
import threading

sum = 0
num = 1000000
lock = threading.Lock()

# lock variable to be accessd by one at the same time
def add(num):
    global sum
    for i in range(num):
        lock.acquire()
        sum += 1
        lock.release()
    return sum

def miu(num):
    global sum
    for i in range(num):
        lock.acquire()
        sum -= 1
        lock.release()
    return sum


def lock_calcu():
    ld1 = threading.Thread(target=add,args=(num,))
    ld2 = threading.Thread(target=miu,args=(num,))

    ld1.start()
    ld2.start()

    ld1.join()
    ld2.join()

if __name__ == "__main__":
    lock_calcu()
    print(sum)

thread safe

if one variable to be used by multi threads at the same time without lock variable but no problem happend, it is called thread safe.

not thread safe: list, set, dict

thread safe: queue

deadlock

two lock both not release, lead to both waitting, deadlock appeard, leading program dead

muiti processes

we know multi threads will cause problem usually, but there are several method more useful than multi threads:

  • subprocess
  • multiprocessing
  • concurrent.futures

iterable/iterator

difference:
iterable -> directly used by for loop
iterator -> not only used by for loop but also can be used by next function

how to distinct each other:

1
2
3
4
5
6
from collections import Iterable,Iterator

lt = [i for i in range(10)]

print(isinstance(lt,Iterable))
print(isinstance(lt,Iterator))

convert iterable to iterator:

1
2
3
4
5
6
7
8
9
10
from collections import Iterable,Iterator

lt = [i for i in range(10)]

print(isinstance(lt,Iterable))
print(isinstance(lt,Iterator))

clt = iter(lt)
print(isinstance(clt,Iterable))
print(isinstance(clt,Iterator))

generator

feature:

  • product data which used by for loop
  • at the end of generator raise StopIterationError
  • can be used by next() function

why we need generator?
all data are calculated step by step when in need, don’t store in memory, use cpu instead of memory. range() is a generator,so we need use it often, it is more useful than list.

how to make a generator

  • ()
  • yield: yield can stop loop until next iterate

use ()

1
2
3
4
5
6
lt = [i for i in range(10)]
lg = (i for i in range(10))

print(type(lt))
# lg is a generator
print(type(lg))

use yield

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
# normal usage
# 1,1,2,3,5,8,13
# def fib(num):
#     a,b,n = 0,1,0
#     while n < num:
#         a,b = b,a+b
#         n += 1
#     return a
# fib(6)


# use yield
def fib(num):
    a,b,n = 0,1,0
    while n < num:
        # yield can stop loop until next iterate
        yield b
        a,b = b,a+b
        n += 1

# generator is iterable
for i in fib(7):
    print(i)

coroutines

coroutines is much more useful than multi threads, it’s need less resource, we can regard it as a generator or a function which can be stopd

how to realize it?

  • yield: return result
  • send: send args to generator
  • yield from: as a middle hand transfer message

yield x: return x
x = yield: get x value from send() function

1
2
3
4
5
6
7
8
9
10
def test_coroutine():
    print("start")
    x = yield
    print("get",x)
    yield x


co = test_coroutine()
next(co)
co.send(20)

yield from

1
2
3
4
5
6
7
8
def test_coroutine(st):
    for i in range(st):
        yield i

def get_num():
    yield from "1234567"

print(list(get_num()))

async/wait

1
2
3
4
5
6
7
8
9
10
11
import asyncio

async def get_num(num):
    for i in range(num):
        print(i)
        await asyncio.sleep(1)

loop = asyncio.get_event_loop()
lt = [get_num(10),get_num(20)]
loop.run_until_complete(asyncio.wait(lt))
loop.close()

concurrent.futures

  • use subprocess, run several python interpreter. each subprocess can use a core of cpu, all core of cpu can be used 100%
  • use multithreads, as we know python can only use one thread at one time, so it uses the same time as one process, and each of thread can not pull use cpu

ThreadPoolExecutor

although my cpu only have 4 core 8 thread, but each cpu didn’t be full used, it is because python can only run one thread at one time, threadpool is also thread.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
from concurrent.futures import ThreadPoolExecutor

def ts(num):
    sum = 0
    print(threading.current_thread())
    for i in range(num):
        sum += 1
    return sum

pool = ThreadPoolExecutor(max_workers=8)

t1 = pool.submit(ts, 100000000)
t2 = pool.submit(ts, 100000000)
t3 = pool.submit(ts, 100000000)
t4 = pool.submit(ts, 100000000)
t5 = pool.submit(ts, 100000000)
t6 = pool.submit(ts, 100000000)
t7 = pool.submit(ts, 100000000)
t8 = pool.submit(ts, 100000000)

print(t1.result())
print(t2.result())
print(t3.result())
print(t4.result())
print(t5.result())
print(t6.result())
print(t7.result())
print(t8.result())

use map

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from concurrent.futures import ThreadPoolExecutor

def ts(num):
    sum = 0
    print(threading.current_thread())
    for i in range(num):
        sum += 1
    return sum

pool = ThreadPoolExecutor(max_workers=8)


lt = [100000000] * 8
for i in pool.map(ts,lt):
    print(i)

ProcessPoolExecutor

all core of cpu can be used 100%

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
from concurrent.futures import ProcessPoolExecutor

def ts(num):
    sum = 0
    print(threading.current_thread())
    for i in range(num):
        sum += 1
    return sum

pool = ProcessPoolExecutor(max_workers=8)


lt = [100000000] * 8
for i in pool.map(ts,lt):
    print(i)

regular expression

  • there are two methods

    1. compile re to a object

      1
      2
      3
      4
      5
      6
      import re

      s = "Hello, Join. where did you go? 123,hk123,H89Hk"
      p = re.compile("[0-9]{1,}")
      m = p.findall(s)
      print(m)

    2. use re directory

      1
      2
      3
      4
      5
      import re

      s = "Hello, Join. where did you go? 123,hk123,H89Hk"
      m = re.findall("[0-9]{1,}",s)
      print(m)

  • ignore case

1
2
3
4
5
6
import re

s = "Hello, Join. where did you go? 123,hkj123,H89Hjk"
p = re.compile("h",re.I)
m = p.findall(s)
print(m)
  • substitute
1
2
3
4
5
6
7
import re

s = "Hello, Join. where did you go? 123,hkj123,H89Hjk"
p = re.compile("\d+")
t = p.sub("000",s)
print(s)
print(t)
  • catch chinese
1
2
3
4
5
6
7
import re

s = "Hello, 世界. 喜欢你where did you go? 123,hkj123,H89Hjk"
p = re.compile("[\u4c00-\u9fa5]+")
t = p.findall(s)

print(t)
  • catch no chinese
1
2
3
4
5
6
7
import re

s = "Hello, 世界. 喜欢你where did you go? 123,hkj123,H89Hjk"
p = re.compile("[^\u4c00-\u9fa5]+")
t = p.findall(s)

print(t)

Xpath

it is used to find xml file

syntax:

  • /: root node
  • //: two node
  • .: current node
  • ..: father node
  • @: property

net program

network model:

  • 7 layers model
    • Layer 7 - Application
    • Layer 6 - Presentation
    • Layer 5 - Session
    • Layer 4 - Transport
    • Layer 3 - Network
    • Layer 2 - Data Link
    • Layer 1 - Physical
  • 4 layers model
    • Layer 4 - Application
    • Layer 3 - Transport
    • Layer 2 - Network
    • Layer 1 - Link

ftp

  • leaning website: link

  • Python ftplib
    python ftplib is a module that implements the client side of the FTP protocol. It contains an FTP client class and some helper functions.

  • Python FTP clas
    The ftplib.FTP() creates a new instance of the FTP class. When host is given, a connection to the host is made with the connect() method.

example:

in my archlinux, i start vsftpd.service first

1
sudo systemctl start vsftpd

and then change the default config /etc/vsftpd.config to enable local user login

at last i write a ftp python file

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
import ftplib

# a anonymous function print lines
line = lambda x: print(x * 80)

# get ftp login user and passwd
user = input("ftp login user name: ")
pwd = input("password: ")

with ftplib.FTP("localhost") as ftp:
    try:
        # get connection
        ftp.login(user=user, passwd=pwd)

        # print welcome information
        info = ftp.getwelcome()
        print("welcome: ",info)
        line("-")
        
        # change current directory
        ftp.cwd(dirname="Desktop")

        # get current directory
        current_dir = ftp.pwd()
        print("current_dir: ",current_dir)
        line("*")

        # get context list
        files = []
        ftp.dir(files.append)
        for i in files:
            print(i)
        line("=")

        # download file, by default the file will be stored at /tmp/
        file_copy = "test.txt"
        with open(file_copy, "w") as f:
            res = ftp.retrlines("RETR "+"1.py", f.write)
            
            if not res.startswith('226 Transfer complete'):
                print('Download failed')
                if os.path.isfile(file_copy):
                    os.remove(file_copy)  
            else:
                print("down")
        
    except ftplib.all_errors as e:
        print(e)

mail

how it works?
process: user mail —-> mail proxy —–> mail server —–> user
agent: ————>mua————–>mta—————>mda
protocal: —————–>SMTP————->POP3/IMAP

MUA: MailUserAgent, user send mail to proxy
MTA: MailTransferAgent, catch mail from proxy to mail server
MDA: MailDeliveryAgent, deliver mail from server to user

how to use it:
there are two modules

  1. email: used to modify mail
  2. smtplib: used to send mail

the following is a simple text mail example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
import smtplib
from email.mime.text import MIMEText


# define some variables
from_addr = "1025096440@qq.com"
# this is a code from web mail
from_pwd = "nrbnnibhjrgrbcdd"
to_addr = "1025096440@qq.com"
smtp_srv = "smtp.qq.com"

# write message
text_msg = MIMEText("第一封邮件", "plain", "utf-8")

# add header
text_msg['From'] = from_addr
text_msg['To'] = to_addr
text_msg['Subject'] = "测试"


try:
    # server address need encode to byte,so we need encode it
    # port 465 is a security port which server accept
    srv = smtplib.SMTP_SSL(smtp_srv.encode(), 465)

    # login
    srv.login(from_addr,from_pwd)

    # send text mail
    srv.sendmail(from_addr, [to_addr], text_msg.as_string())
    srv.quit()
except Exception as e:
    print(e)
else:
    print("done")

the following is a text format mail with additional file 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEBase,MIMEMultipart



# some variables
from_addr = "1025096440@qq.com"
# this is a code from web mail
from_pwd = "nrbnnibhjrgrbcdd"
to_addr = "1025096440@qq.com"
smtp_srv = "smtp.qq.com"
files = r"/home/narcissus/Desktop/1.py"

# initial multipart object
mail_mul = MIMEMultipart()
# add header
mail_mul['From'] = from_addr
mail_mul['To'] = to_addr
mail_mul['Subject'] = "use python email and smtplib module send mail"

# write message
text_msg = MIMEText("the email include text message and file", "plain", "utf-8")

# add text to multipart object
mail_mul.attach(text_msg)

# add file
with open(files, "rb") as f:
    line = f.read()
    content = MIMEText(line, "base64", "utf-8")
    content["Content-type"] = "application/octet-stream"
    content["Content-Disposition"] = "attachment; filename=files"
    mail_mul.attach(content)

# send mail
try:
    # server address need encode to byte,so we need encode it
    # port 465 is a security port which server accept
    srv = smtplib.SMTP_SSL(smtp_srv.encode(), 465)

    # login
    srv.login(from_addr,from_pwd)

    # send text mail
    srv.sendmail(from_addr, [to_addr], mail_mul.as_string())
    srv.quit()
except Exception as e:
    print(e)
else:
    print("done")

the following is a html format mail example.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
import smtplib
from email.mime.text import MIMEText

# write message
html_msg = """
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta http-equiv="X-UA-Compatible" content="ie=edge">
        <title>Document</title>
        <script src="./vue.js" charset="utf-8"></script>
    </head>
    <body>
        <h1 style="background: #7fedbb">有情人难成属眷</h1>
    </body>
    </html>
    """
html_msg = MIMEText(html_msg, "html", "utf-8")
# some variables
from_addr = "1025096440@qq.com"
# this is a code from web mail
from_pwd = "nrbnnibhjrgrbcdd"
to_addr = "1025096440@qq.com"
smtp_srv = "smtp.qq.com"

try:
    # server address need encode to byte,so we need encode it
    # port 465 is a security port which server accept
    srv = smtplib.SMTP_SSL(smtp_srv.encode(), 465)

    # login
    srv.login(from_addr,from_pwd)

    # send html mail
    srv.sendmail(from_addr, [to_addr], html_msg.as_string())
    srv.quit()
except Exception as e:
    print(e)
else:
    print("done")

alternative choose html format or text format, because some fo device can not display html format

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
import smtplib
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart



# some variables
from_addr = "1025096440@qq.com"
# this is a code from web mail
from_pwd = "nrbnnibhjrgrbcdd"
to_addr = "1025096440@qq.com"
smtp_srv = "smtp.qq.com"

# initial multipart object
mail_mul = MIMEMultipart("alternative")
# add header
mail_mul['From'] = from_addr
mail_mul['To'] = to_addr
mail_mul['Subject'] = "use python email and smtplib module send mail"

# write message
text_msg = MIMEText("alternative choose text or html style", "plain", "utf-8")

# add text to multipart object
mail_mul.attach(text_msg)

# write html
html_msg = """
    <!DOCTYPE html>
    <html lang="en">
    <head>
        <meta charset="UTF-8">
        <meta name="viewport" content="width=device-width, initial-scale=1.0">
        <meta http-equiv="X-UA-Compatible" content="ie=edge">
        <title>Document</title>
        <script src="./vue.js" charset="utf-8"></script>
    </head>
    <body>
        <h1 style="background: #7fedbb">有情人难成属眷</h1>
    </body>
    </html>
    """
html_msg = MIMEText(html_msg, "html", "utf-8")
mail_mul.attach(html_msg)

# send mail
try:
    # server address need encode to byte,so we need encode it
    # port 465 is a security port which server accept
    srv = smtplib.SMTP_SSL(smtp_srv.encode(), 465)

    # login
    srv.login(from_addr,from_pwd)

    # send text mail
    srv.sendmail(from_addr, [to_addr], mail_mul.as_string())
    srv.quit()
except Exception as e:
    print(e)
else:
    print("done")

the end! I am really happy that i learn all of them. It is really awesome,nice.