Mọi mã thông báo được tạo bởi trình mã thông báo đều có một loại. Các loại này được biểu diễn bằng các hằng số nguyên. Giá trị số nguyên thực tế của hằng số mã thông báo là không quan trọng (ngoại trừ ) và không bao giờ được sử dụng hoặc dựa vào. Thay vào đó, hãy tham khảo mã thông báo theo tên biến của chúng và sử dụng từ điển >>> import keyword >>> keyword.iskeyword('or') True1 để lấy tên của loại mã thông báo. Chẳng hạn, giá trị số nguyên chính xác có thể thay đổi giữa các phiên bản Python, nếu mã thông báo mới được thêm hoặc xóa (và thực tế, trong các phiên bản Python gần đây, chúng có). Trong các ví dụ bên dưới, số mã thông báo hiển thị ở đầu ra là số từ Python 3. 9 Show
Lý do các loại mã thông báo được trình bày theo cách này là vì mã thông báo thực tế được trình thông dịch Python sử dụng không phải là mô-đun >>> import keyword >>> keyword.iskeyword('or') True2; . C không có hệ thống đối tượng như Python. Thay vào đó, các kiểu liệt kê được biểu diễn bằng các số nguyên (thực ra, >>> import keyword >>> keyword.iskeyword('or') True3 có một mảng lớn các kiểu mã thông báo. Giá trị nguyên của mỗi mã thông báo là chỉ mục của nó trong mảng đó). Mô-đun >>> import keyword >>> keyword.iskeyword('or') True2 được viết bằng Python thuần túy, nhưng các giá trị và tên loại mã thông báo phản ánh các giá trị và tên từ mã thông báo C, với ba trường hợp ngoại lệ. , , và Tất cả các loại mã thông báo được xác định trong mô-đun >>> import keyword >>> keyword.iskeyword('or') True8, nhưng mô-đun >>> import keyword >>> keyword.iskeyword('or') True2 thì có ____6_______0, vì vậy chúng cũng có thể được nhập từ ___________2. Do đó, cách dễ nhất là chỉ cần nhập mọi thứ từ >>> import keyword >>> keyword.iskeyword('or') True2. Hơn nữa, các mã thông báo , và đã nói ở trên không thể nhập được từ >>> import keyword >>> keyword.iskeyword('or') True8 trước Python 3. 7, chỉ từ >>> import keyword >>> keyword.iskeyword('or') True2 Từ điển >>> import keyword >>> keyword.iskeyword('or') True 1Từ điển >>> import keyword >>> keyword.iskeyword('or') True1 ánh xạ các mã thông báo trở lại tên của chúng >>> import tokenize >>> tokenize.STRING 3 >>> tokenize.tok_name[tokenize.STRING] # Can also use token.tok_name 'STRING' Mã thông báoĐể đơn giản hóa các phần bên dưới, hàm tiện ích sau đây được sử dụng cho tất cả các ví dụ >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') 0Đây luôn là mã thông báo cuối cùng được phát ra bởi >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1, trừ khi nó tăng. Các thuộc tính >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2 và >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')3 luôn là >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4. Các dòng >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 và >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 luôn nhiều hơn một dòng so với tổng số dòng trong đầu vào và các cột >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 và >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 luôn bằng 0 Đối với hầu hết các ứng dụng, không cần thiết phải lo lắng rõ ràng về >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')0, vì >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 dừng lặp lại sau khi mã thông báo cuối cùng được tạo ra >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='') >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') 1Loại mã thông báo >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 được sử dụng cho bất kỳ mã định danh Python nào, cũng như mọi từ khóa. là các tên Python được bảo lưu, nghĩa là chúng không thể được gán cho, chẳng hạn như >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')3, >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 và >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') Để biết mã thông báo >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 có phải là từ khóa hay không, hãy sử dụng trên >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2 >>> import keyword >>> keyword.iskeyword('or') True Lưu ý thêm, trong nội bộ, mô-đun >>> import keyword >>> keyword.iskeyword('or') True2 sử dụng phương pháp này để kiểm tra xem mã thông báo có phải là mã thông báo >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 hay không. Các quy tắc đầy đủ cho những gì tạo ra hơi phức tạp, vì chúng liên quan đến một bảng ký tự Unicode lớn. Người ta phải luôn sử dụng phương pháp >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')0 để kiểm tra xem một chuỗi có phải là mã định danh Python hợp lệ hay không, kết hợp với kiểm tra >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')7. Việc kiểm tra xem một chuỗi có phải là mã định danh hay không bằng cách sử dụng các biểu thức chính quy rất không được khuyến khích >>> 'α'.isidentifier() True >>> 'or'.isidentifier() True >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') 4Loại mã thông báo >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 được sử dụng cho bất kỳ chữ số nào, bao gồm chữ số nguyên (thập phân), chữ số nguyên nhị phân, bát phân và thập lục phân, số dấu phẩy động (bao gồm ký hiệu khoa học) và chữ số ảo (như >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6) >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') Lưu ý rằng mặc dù các chữ cái như >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')7 là một loại duy nhất của >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')8, nhưng chúng mã hóa thành >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)00), >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)01 ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)02), >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)04) >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') Chữ số không hợp lệ có thể mã hóa dưới dạng nhiều chữ số >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') Một lợi thế của việc sử dụng >>> import keyword >>> keyword.iskeyword('or') True2 so với >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)06 là các số dấu phẩy động không được làm tròn ở giai đoạn mã thông báo, vì vậy có thể truy cập toàn bộ thông tin đầu vào >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)0 Điều này có thể được sử dụng, ví dụ, để bọc các số dấu phẩy động với một loại hỗ trợ độ chính xác tùy ý, chẳng hạn như >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)07. xem Trong Python >=3. 6, chữ số có thể có , như >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)08 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)1 Trong Trăn 3. 5, điều này sẽ mã hóa dưới dạng hai mã thông báo, >>> print_tokens('012\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0', start=(1, 0), end=(1, 1), line='012\n') TokenInfo(type=2 (NUMBER), string='12', start=(1, 1), end=(1, 3), line='012\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 3), end=(1, 4), line='012\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0x1.0\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0x1', start=(1, 0), end=(1, 3), line='0x1.0\n') TokenInfo(type=2 (NUMBER), string='.0', start=(1, 3), end=(1, 5), line='0x1.0\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0x1.0\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') >>> print_tokens('0o184\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='0o1', start=(1, 0), end=(1, 3), line='0o184\n') TokenInfo(type=2 (NUMBER), string='84', start=(1, 3), end=(1, 5), line='0o184\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='0o184\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)10) và >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)12) (và sẽ không có giá trị về mặt cú pháp trong bất kỳ ngữ cảnh nào) >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)2 Trong phần này, chúng ta sẽ xem cách sử dụng >>> import keyword >>> keyword.iskeyword('or') True2 để chuyển tính năng này sang Python 3. 5 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 14Loại mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)14 khớp với bất kỳ chuỗi ký tự nào, bao gồm chuỗi trích dẫn đơn, chuỗi trích dẫn kép, chuỗi trích dẫn đơn và kép (i. e. , chuỗi nhiều dòng hoặc “docstrings”), raw, “unicode”, byte và f-strings (Python 3. 6+) >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)3 Lưu ý rằng mặc dù Python hoàn toàn nối các chuỗi ký tự, nhưng >>> import keyword >>> keyword.iskeyword('or') True2 mã hóa chúng một cách riêng biệt >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)4 Trong trường hợp chuỗi thô, “unicode”, byte và chuỗi f, tiền tố chuỗi được bao gồm trong chuỗi được mã hóa >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)5 chuỗi f (Python 3. 6+) được phân tích thành một mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)14 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)6 Hàm >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)18 có thể được sử dụng để phân tích các chuỗi định dạng (bao gồm cả chuỗi f) >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)7 Để lấy giá trị chuỗi từ một chuỗi ký tự được mã hóa (i. e. , để loại bỏ các ký tự trích dẫn), hãy sử dụng >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)19. Điều này được khuyến nghị thay vì cố gắng loại bỏ các dấu ngoặc kép theo cách thủ công, dễ bị lỗi hoặc sử dụng >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)20 thô, có thể thực thi mã tùy ý trong trường hợp chuỗi f >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)8 Hành vi lỗiNếu một chuỗi được trích dẫn không được mở, thì dấu phân cách chuỗi mở được mã hóa thành và phần còn lại được mã hóa như thể nó không có trong một chuỗi >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)9 Hành vi này có thể hữu ích để xử lý các tình huống lỗi. Ví dụ: nếu bạn định xây dựng một công cụ đánh dấu cú pháp bằng cách sử dụng >>> import keyword >>> keyword.iskeyword('or') True2, bạn có thể không nhất thiết muốn một chuỗi không được tiết lộ để đánh dấu phần còn lại của tài liệu dưới dạng một chuỗi (những thứ như vậy phổ biến trong môi trường chỉnh sửa “trực tiếp”) Tuy nhiên, nếu một chuỗi trích dẫn ba (i. e. , chuỗi nhiều dòng hoặc “docstring”) không được đóng, >>> import keyword >>> keyword.iskeyword('or') True2 sẽ tăng khi đạt đến nó >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')0 Hành vi này có thể gây phiền nhiễu để giải quyết trong thực tế. Đối với nhiều ứng dụng, cách chính xác để xử lý tình huống này là xem xét rằng vì chuỗi không đóng là nhiều dòng, nên phần còn lại của đầu vào từ nơi ký tự được nâng lên nằm bên trong chuỗi không đóng. Lưu ý cuối cùng, hãy lưu ý rằng có thể xây dựng các chuỗi ký tự mã hóa mà không có bất kỳ lỗi nào, nhưng tăng >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)26 khi được trình thông dịch phân tích cú pháp >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 Để kiểm tra xem một chuỗi ký tự có hợp lệ hay không, bạn có thể sử dụng hàm >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)19, đây là hàm an toàn để sử dụng cho đầu vào không đáng tin cậy >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 28Loại mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)28 đại diện cho một ký tự xuống dòng ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)30 hoặc >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)31) kết thúc một dòng logic của mã Python. Các dòng mới không kết thúc một dòng logic sử dụng mã Python >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')3 Các dòng mới kiểu Windows ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)31) được mã hóa thành một mã thông báo duy nhất >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 Bắt đầu từ Python 3. 6. 7 và 3. 7. 1 bản phát hành bản vá, >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)28 được phát ra ở cuối tất cả đầu vào mã thông báo ngay cả khi nó không kết thúc bằng một dòng mới (ngoại trừ các dòng chỉ là nhận xét). Thay đổi này được thực hiện để thống nhất với tokenize nội bộ. mô-đun c được sử dụng bởi chính Python. Các >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)28 ngầm định có ____15_______3 được đặt thành ____15_______4. Thay đổi này không được thực hiện đối với Python 3. 5, đã có trong. Các ví dụ trong tài liệu này đều sử dụng 3. 6. 7+ hành vi. Nếu muốn có sự nhất quán, người ta luôn có thể buộc đầu vào kết thúc bằng một dòng mới (đây là lý do tại sao mọi ví dụ trong tài liệu này đều có một dòng mới ở cuối) >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 38>>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 39Loại mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 đại diện cho thụt đầu dòng cho các khối được thụt lề. Bản thân phần thụt lề (văn bản từ đầu dòng đến ký tự không phải khoảng trắng đầu tiên) nằm trong thuộc tính >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2. >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 được phát ra một lần trên mỗi khối văn bản thụt lề, không phải một lần trên mỗi dòng Loại mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)39 đại diện cho một vết lõm. Mỗi mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 được khớp với mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)39 tương ứng. Thuộc tính >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2 của >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)39 luôn là >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4. Vị trí >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 và >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 của mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)39 là vị trí đầu tiên trong dòng sau dấu thụt (ngay cả khi có nhiều >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)39 liên tiếp) Xét ví dụ giả sau >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 Có một >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 trước khối >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)54, một >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 trước khối >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)56 và hai >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)57 trước khối >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)58 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 không được sử dụng để thụt lề trên phần tiếp theo của dòng >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 Thụt đầu dòng có thể là bất kỳ số khoảng trắng hoặc tab nào. Hạn chế duy nhất là mọi mức thụt lề không thụt vào phải khớp với mức thụt lề ngoài trước đó. Nếu một vết lõm không khớp với mức độ thụt lề bên ngoài, >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 sẽ tăng >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')7 Mức độ thụt đầu dòng tại một điểm cụ thể trong luồng mã thông báo có thể được xác định bằng cách tăng và giảm bộ đếm cho mỗi mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 và >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)39 hoặc nếu khoảng cách thụt đầu dòng chính xác được yêu cầu, bằng cách duy trì một ngăn xếp chuỗi >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)38 ( mà Python cho phép khác nhau . xem >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 65>>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 66Mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)65 và >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)66 được mã hóa thành. Tuy nhiên, do một lỗi xuất hiện trong các phiên bản Python trước 3. 7, thuộc tính >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)70 của các mã thông báo này sẽ là >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)01 thay vì loại chính xác >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')8 Lỗi này đã được sửa trong Python 3. 7 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')9 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 01>>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)01 là loại mã thông báo chung cho tất cả các toán tử, dấu phân cách và dấu chấm lửng bằng chữ. Điều này không bao gồm các ký tự và toán tử không được trình phân tích cú pháp nhận dạng (chúng được phân tích cú pháp dưới dạng ) Khi sử dụng >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1, loại mã thông báo cho một toán tử, dấu phân cách hoặc mã thông báo bằng dấu chấm lửng sẽ là >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)01. Để có được loại mã thông báo chính xác, hãy sử dụng thuộc tính >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)70 của têntuple. >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)78 tương đương với >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)79 đối với các loại mã thông báo còn lại (với hai trường hợp ngoại lệ, xem ghi chú bên dưới) >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')0 Bảng sau đây liệt kê tất cả các loại >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)01 chính xác và các ký tự tương ứng của chúng Loại mã thông báo chính xác Chuỗi giá trị >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)81 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)82 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)83 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)84 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)85 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)86 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)87 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)88 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)89 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)90 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)91 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)92 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)93 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)94 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)95 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)02 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)97 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)98 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)99 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')00 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')01 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')02 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')03 ` >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')04 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')05 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')06 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')07 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')08 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')09 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')10 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')11 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')12 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')13 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')14 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')15 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')16 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')17 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')18 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')19 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')20 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')21 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')22 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')23 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')24 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')25 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')26 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')27 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')28 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')29 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')30 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')31 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')32 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')33 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')34 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')35 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')36 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')37 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')38 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')39 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')40 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')41 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')42 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')43 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')44 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')45 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')46 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')47 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')48 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')49 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')50 ` >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')51 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')52 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')53 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')54 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')55 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')56 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')57 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')58 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')59 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')60 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')61 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')62 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')63 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')64 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')65 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')66 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)65 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')68 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)66 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')70 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')71 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')72 >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') 73>>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='') 74Các loại mã thông báo >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')73 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')74 được sử dụng để mã hóa các từ khóa >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')77 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 trong Python 3. 5 và 3. 6. Chúng không tồn tại trong Python 3. 7+ Trong Trăn 3. 5 và 3. 6, >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')77 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 là từ khóa giả. Để hỗ trợ quá trình chuyển đổi trong việc bổ sung các từ khóa mới, >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')77 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 được giữ làm tên biến hợp lệ bên ngoài khối >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')83 >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')1 Để hỗ trợ điều này, trong Python 3. 5 và 3. 6 khi gặp phải >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')83, >>> import keyword >>> keyword.iskeyword('or') True2 theo dõi mức thụt đầu dòng của nó và tất cả mã thông báo >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')77 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 được lồng bên dưới nó được mã hóa lần lượt là >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')73 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')74 (bao gồm cả mã thông báo >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 từ >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')83). Mặt khác, >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')77 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 được mã hóa thành >>> print_tokens('1+2j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='1', start=(1, 0), end=(1, 1), line='1+2j\n') TokenInfo(type=54 (OP), string='+', start=(1, 1), end=(1, 2), line='1+2j\n') TokenInfo(type=2 (NUMBER), string='2j', start=(1, 2), end=(1, 4), line='1+2j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 4), end=(1, 5), line='1+2j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1, như trong ví dụ trên >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')2 Trong Trăn 3. 7, >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')78 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')77 là các từ khóa phù hợp và được mã hóa giống như tất cả các từ khóa khác. Trong Trăn 3. 7, các loại mã thông báo >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')73 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')74 đã bị xóa khỏi mô-đun >>> import keyword >>> keyword.iskeyword('or') True8 >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')3 Trong Trăn 3. 8, mã thông báo >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')74 và >>> print_tokens('x + 1\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='x', start=(1, 0), end=(1, 1), line='x + 1\n') TokenInfo(type=54 (OP), string='+', start=(1, 2), end=(1, 3), line='x + 1\n') TokenInfo(type=2 (NUMBER), string='1', start=(1, 4), end=(1, 5), line='x + 1\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 5), end=(1, 6), line='x + 1\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')73 đã được thêm lại vào mô-đun >>> import keyword >>> keyword.iskeyword('or') True8, nhưng chúng không được mã hóa theo mặc định (hành vi giống như trong 3. 7). Họ ở đó chỉ tạo điều kiện cho cờ >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')04 mới thành >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')05 cho phép phân tích cú pháp Python như các phiên bản cũ hơn >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='') 06>>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='') 07>>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')06 và >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')07 được bao gồm ở đây cho đầy đủ, vì chúng có trong từ điển. Chúng được sử dụng trong mã thông báo C để mã hóa nhận xét loại, nhưng mã thông báo Python chưa mã hóa chúng. Điều này có lẽ sẽ thay đổi trong tương lai, vì mã thông báo Python thường được tạo để có hành vi giống như mã thông báo C. Nó đã được thêm vào Python 3. 8 >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')4 >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='') 11>>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')11 cũng được bao gồm ở đây cho đầy đủ. Nó có trong từ điển, nhưng nó không thực sự được sử dụng ở bất kỳ đâu trong mô-đun tokenize và không thể được phát ra dưới dạng mã thông báo từ >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1. Nó được sử dụng trong mã thông báo C như một phần của trình phân tích cú pháp PEG để xử lý “từ khóa mềm” như >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')15 và >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')16 chỉ là từ khóa khi được sử dụng trong các ngữ cảnh nhất định. Nó đã được thêm vào Python 3. 10 >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')5 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok) 21Loại >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)21 được sử dụng cho bất kỳ ký tự nào không được nhận dạng. Các đầu vào mã hóa các >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)21 không thể là Python hợp lệ, nhưng loại mã thông báo này được sử dụng để các ứng dụng xử lý mã thông báo có thể thực hiện khôi phục lỗi, vì phần còn lại của luồng đầu vào được mã hóa bình thường. Nó cũng có thể được sử dụng để xử lý các phần mở rộng cho cú pháp Python (xem tài liệu). Mỗi ký tự không được nhận dạng được mã hóa riêng >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')6 >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)21 cũng được sử dụng cho các chuỗi trích dẫn đơn không được đóng trước một dòng mới. Xem phần để biết thêm thông tin >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)9 Nếu chuỗi được tiếp tục và không được đóng lại, toàn bộ chuỗi sẽ được mã hóa dưới dạng mã thông báo lỗi. Mặt khác, chỉ dấu phân cách trích dẫn bắt đầu là >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')8 Trong trường hợp các chuỗi trích dẫn đơn lẻ không được đóng tiếp tục, các khoảng trắng trước chuỗi cũng được mã hóa thành >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)21 >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')9 Điều này không áp dụng cho các chuỗi tiếp tục không được đóng >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')0 Do đó, mã xử lý >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)21 cụ thể cho các chuỗi không được đóng nên kiểm tra >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')24 >>> import keyword >>> keyword.iskeyword('or') True 5Loại mã thông báo >>> import keyword >>> keyword.iskeyword('or') True5 đại diện cho nhận xét. Nếu một nhận xét kéo dài trên nhiều dòng, thì mỗi dòng sẽ được mã hóa riêng >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1 Loại mã thông báo >>> import keyword >>> keyword.iskeyword('or') True5 chỉ tồn tại trong thư viện chuẩn Triển khai Python của >>> import keyword >>> keyword.iskeyword('or') True2. Việc triển khai C được trình thông dịch sử dụng bỏ qua các nhận xét. Trong các phiên bản Python trước 3. 7, >>> import keyword >>> keyword.iskeyword('or') True5 chỉ có thể nhập được từ mô-đun >>> import keyword >>> keyword.iskeyword('or') True2. Trong 3. 7, nó cũng được thêm vào mô-đun >>> import keyword >>> keyword.iskeyword('or') True8 >>> import keyword >>> keyword.iskeyword('or') True 6Loại mã thông báo >>> import keyword >>> keyword.iskeyword('or') True6 đại diện cho các ký tự dòng mới ( >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)30 hoặc >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)31) không kết thúc các dòng mã logic. Các dòng mới kết thúc các dòng logic của vùng mã Python được mã hóa bằng loại mã thông báo Có hai tình huống mà các dòng mới được mã hóa là >>> import keyword >>> keyword.iskeyword('or') True6
Lưu ý rằng các dòng mới được thoát (trước >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')38) được coi là khoảng trắng, nghĩa là chúng hoàn toàn không mã hóa. Do đó, bạn phải luôn sử dụng số dòng trong thuộc tính >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 và >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 của bộ tên >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')41. Không bao giờ cố gắng xác định số dòng bằng cách đếm và >>> import keyword >>> keyword.iskeyword('or') True6 mã thông báo >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')4 Loại mã thông báo >>> import keyword >>> keyword.iskeyword('or') True6 chỉ tồn tại trong thư viện chuẩn Triển khai Python của >>> import keyword >>> keyword.iskeyword('or') True2. Việc triển khai C được sử dụng bởi trình thông dịch chỉ có. Trong các phiên bản Python trước 3. 7, >>> import keyword >>> keyword.iskeyword('or') True6 chỉ có thể nhập được từ mô-đun >>> import keyword >>> keyword.iskeyword('or') True2. Trong 3. 7, nó cũng được thêm vào mô-đun >>> import keyword >>> keyword.iskeyword('or') True8 >>> import keyword >>> keyword.iskeyword('or') True 7>>> import keyword >>> keyword.iskeyword('or') True7 là một loại mã thông báo đặc biệt đại diện cho việc mã hóa đầu vào. Nó luôn là mã thông báo đầu tiên được phát ra bởi >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')1, trừ khi mã hóa được phát hiện không hợp lệ, trong trường hợp đó, nó sẽ tăng. Mã hóa được phát hiện thông qua nhận xét được định dạng PEP 263 ở một trong hai dòng đầu tiên của đầu vào (như >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')54; các nhận xét như vậy vẫn được mã hóa dưới dạng a) hoặc ký tự Unicode BOM Mã hóa được phát hiện nằm trong thuộc tính >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2 của >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')41. >>> import keyword >>> keyword.iskeyword('or') True7 là loại mã thông báo duy nhất mà >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')59 không xuất hiện theo nghĩa đen trong đầu vào. Mã hóa mặc định là >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')60 Nếu bạn chỉ muốn phát hiện mã hóa và không muốn gì khác, hãy sử dụng. Nếu bạn chỉ cần mã hóa để chuyển đến >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')62, hãy sử dụng Số dòng và cột của >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 và >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 cho >>> import keyword >>> keyword.iskeyword('or') True7 sẽ luôn là >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')67 >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')5 Mã thông báo >>> import keyword >>> keyword.iskeyword('or') True7 thường không được sử dụng khi xử lý mã thông báo, mặc dù sự hiện diện được đảm bảo của nó với tư cách là mã thông báo đầu tiên có thể hữu ích khi xử lý mã thông báo vì nó sẽ đảm bảo rằng mọi loại mã thông báo khác sẽ có một số mã thông báo trước đó từ luồng (xem phần ) Nói một cách chính xác, >>> print_tokens('10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=2 (NUMBER), string='10', start=(1, 0), end=(1, 2), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 3), end=(1, 4), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0b101', start=(1, 5), end=(1, 10), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 11), end=(1, 12), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0o10', start=(1, 13), end=(1, 17), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 18), end=(1, 19), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='0xa', start=(1, 20), end=(1, 23), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='-', start=(1, 24), end=(1, 25), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1.0', start=(1, 26), end=(1, 29), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 30), end=(1, 31), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1e1', start=(1, 32), end=(1, 35), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=54 (OP), string='+', start=(1, 36), end=(1, 37), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=2 (NUMBER), string='1j', start=(1, 38), end=(1, 40), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 40), end=(1, 41), line='10 + 0b101 + 0o10 + 0xa - 1.0 + 1e1 + 1j\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')2 của mọi mã thông báo trong luồng mã thông báo phải có thể giải mã được bằng cách mã hóa mã thông báo >>> import keyword >>> keyword.iskeyword('or') True7 (e. g. , nếu mã hóa là >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')71, thì mã thông báo không được bao gồm bất kỳ ký tự không phải ASCII nào) Loại mã thông báo >>> import keyword >>> keyword.iskeyword('or') True7 chỉ tồn tại trong thư viện chuẩn Triển khai Python của >>> import keyword >>> keyword.iskeyword('or') True2. Việc triển khai C được trình thông dịch sử dụng sẽ phát hiện mã hóa riêng. Trong các phiên bản Python trước 3. 7, >>> import keyword >>> keyword.iskeyword('or') True7 chỉ có thể nhập được từ mô-đun >>> import keyword >>> keyword.iskeyword('or') True2. Trong 3. 7, nó cũng được thêm vào mô-đun >>> import keyword >>> keyword.iskeyword('or') True8 >>> import keyword >>> keyword.iskeyword('or') True 0Số loại mã thông báo (không bao gồm hoặc chính nó) Trong Trăn 3. 5 và 3. 6, >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')79 và >>> print_tokens('') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=0 (ENDMARKER), string='', start=(1, 0), end=(1, 0), line='')80 khác nhau, vì , , và nằm trong >>> import keyword >>> keyword.iskeyword('or') True2 nhưng không nằm trong >>> import keyword >>> keyword.iskeyword('or') True8. Trong các phiên bản này, >>> import keyword >>> keyword.iskeyword('or') True0 cũng không có trong từ điển >>> import keyword >>> keyword.iskeyword('or') True1 Giá trị của >>> import keyword >>> keyword.iskeyword('or') True0 khác nhau giữa các phiên bản Python. Trăn 3. 7 đã xóa và mã thông báo. Trăn 3. 8 đã thêm các mã thông báo mới , , và , và được thêm lại và. Trong Trăn 3. 10, đã được thêm vào >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')6 >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')7 >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')8 >>> print_tokens('a or α\n') TokenInfo(type=63 (ENCODING), string='utf-8', start=(0, 0), end=(0, 0), line='') TokenInfo(type=1 (NAME), string='a', start=(1, 0), end=(1, 1), line='a or α\n') TokenInfo(type=1 (NAME), string='or', start=(1, 2), end=(1, 4), line='a or α\n') TokenInfo(type=1 (NAME), string='α', start=(1, 5), end=(1, 6), line='a or α\n') TokenInfo(type=4 (NEWLINE), string='\n', start=(1, 6), end=(1, 7), line='a or α\n') TokenInfo(type=0 (ENDMARKER), string='', start=(2, 0), end=(2, 0), line='')9 [ 1 ] (,) Do có lỗi, mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)70 cho mã thông báo >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)65 và >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)66 là >>> import io >>> def print_tokens(s): .. for tok in tokenize.tokenize(io.BytesIO(s.encode('utf-8')).readline): .. print(tok)01 trong các phiên bản Python trước 3. 7. Thấy Có bao nhiêu mã thông báo trong Python?Có 5 loại mã thông báo trong python được liệt kê bên dưới. từ khóa. định danh. chữ.
Mã thông báo và các loại của nó trong Python là gì?Mã thông báo. Python ngắt từng dòng logic thành một chuỗi các thành phần từ vựng cơ bản được gọi là mã thông báo. Mỗi mã thông báo tương ứng với một chuỗi con của dòng logic. Các loại mã thông báo thông thường là số nhận dạng, từ khóa, toán tử, dấu phân cách và ký tự , như được đề cập trong các phần sau.
Mã thông báo trong Python là gì cho 2 ví dụ?Chúng được sử dụng cho các tính năng đặc biệt của chúng. Trong Python, chúng tôi có 33 từ khóa, một số trong số đó là. thử, Sai, Đúng, lớp, ngắt, tiếp tục và, như, khẳng định, trong khi, cho, trong, nâng cao, ngoại trừ, hoặc, không, nếu, elif, in, nhập, v.v. |
