python代码词频分析（python实现词法分析器）
时间：2022-03-31 12:17:33 类别：脚本大全 浏览量：623
python代码词频分析
python实现词法分析器简单python词法分析器实现，供大家参考，具体内容如下
词法分析器状态转换图：
词法分析器总流程图：
预处理程序：
词法分析器：
词法分析器程序详细设计
详细代码实现:
|
|
#!/usr/bin/env python3.4# coding=utf-8import sysimport stringkeywards = {}# 关键字部分keywards['false'] = 101keywards['class'] = 102keywards['finally'] = 103keywards['is'] = 104keywards['return'] = 105keywards['none'] = 106keywards['continue'] = 107keywards['for'] = 108keywards['lambda'] = 109keywards['try'] = 110keywards['true'] = 111keywards['def'] = 112keywards['from'] = 113keywards['nonlocal'] = 114keywards['while'] = 115keywards['and'] = 116keywards['del'] = 117keywards['global'] = 118keywards['not'] = 119keywards['with'] = 120keywards['as'] = 121keywards['elif'] = 122keywards['if'] = 123keywards['or'] = 124keywards['yield'] = 125keywards['assert'] = 126keywards['else'] = 127keywards['import'] = 128keywards['pass'] = 129keywards['break'] = 130keywards['except'] = 131keywards['in'] = 132keywards['raise'] = 133# 符号keywards['+'] = 201keywards['-'] = 202keywards['*'] = 203keywards['/'] = 204keywards['='] = 205keywards[':'] = 206keywards['<'] = 207keywards['>'] = 208keywards['%'] = 209keywards['&'] = 210keywards['!'] = 211keywards['('] = 212keywards[')'] = 213keywards['['] = 214keywards[']'] = 215keywards['{'] = 216keywards['}'] = 217keywards['#'] = 218keywards['|'] = 219keywards[','] = 220# 变量# keywards['var'] = 301# 常量# keywards['const'] = 401# error# keywards['const'] = 501signlist = {}# 预处理函数，将文件中的空格，换行等无关字符处理掉def pretreatment(file_name): try: fp_read = open(file_name, 'r') fp_write = open('file.tmp', 'w') sign = 0 while true: read = fp_read.readline() if not read: break length = len(read) i = -1 while i < length - 1: i += 1 if sign == 0: if read[i] == ' ': continue if read[i] == '#': break elif read[i] == ' ': if sign == 1: continue else: sign = 1 fp_write.write(' ') elif read[i] == '\t': if sign == 1: continue else: sign = 1 fp_write.write(' ') elif read[i] == '\n': if sign == 1: continue else: fp_write.write(' ') sign = 1 elif read[i] == '"': fp_write.write(read[i]) i += 1 while i < length and read[i] != '"': fp_write.write(read[i]) i += 1 if i >= length: break fp_write.write(read[i]) elif read[i] == "'": fp_write.write(read[i]) i += 1 while i < length and read[i] != "'": fp_write.write(read[i]) i += 1 if i >= length: break fp_write.write(read[i]) else: sign = 3 fp_write.write(read[i]) except exception: print(file_name, ': this filename not found!')def save(string): if string in keywards.keys(): if string not in signlist.keys(): signlist[string] = keywards[string] else: try: float(string) save_const(string) except valueerror: save_var(string)def save_var(string): if string not in signlist.keys(): if len(string.strip()) < 1: pass else: if is_signal(string) == 1: signlist[string] = 301 else: signlist[string] = 501def save_const(string): if string not in signlist.keys(): signlist[string] = 401def save_error(string): if string not in signlist.keys(): signlist[string] = 501def is_signal(s): if s[0] == '_' or s[0] in string.ascii_letters: for i in s: if i in string.ascii_letters or i == '_' or i in string.digits: pass else: return 0 return 1 else: return 0def recognition(filename): try: fp_read = open(filename, 'r') string = "" sign = 0 while true: read = fp_read.read(1) if not read: break if read == ' ': if len(string.strip()) < 1: sign = 0 pass else: if sign == 1 or sign == 2: string += read else: save(string) string = "" sign = 0 elif read == '(': if sign == 1 or sign == 2: string += read else: save(string) string = "" save('(') elif read == ')': if sign == 1 or sign == 2: string += read else: save(string) string = "" save(')') elif read == '[': if sign == 1 or sign == 2: string += read else: save(string) string = "" save('[') elif read == ']': if sign == 1 or sign == 2: string += read else: save(string) string = "" save(']') elif read == '{': if sign == 1 or sign == 2: string += read else: save(string) string = "" save('{') elif read == '}': if sign == 1 or sign == 2: string += read else: save(string) string = "" save('}') elif read == '<': save(string) string = "" save('<') elif read == '>': save(string) string = "" save('>') elif read == ',': save(string) string = "" save(',') elif read == "'": string += read if sign == 1: sign = 0 save_const(string) string = "" else: if sign != 2: sign = 1 elif read == '"': string += read if sign == 2: sign = 0 save_const(string) string = "" else: if sign != 1: sign = 2 elif read == ':': if sign == 1 or sign == 2: string += read else: save(string) string = "" save(':') elif read == '+': save(string) string = "" save('+') elif read == '=': save(string) string = "" save('=') else: string += read except exception as e: print(e)def main(): if len(sys.argv) < 2: print("please input filename") else: pretreatment(sys.argv[1]) recognition('file.tmp') for i in signlist.keys(): print("(", signlist[i], ",", i, ")")if __name__ == '__main__': main() |
以上就是本文的全部内容，希望对大家的学习有所帮助，也希望大家多多支持开心学习网。
原文链接：https://blog.csdn.net/IT_DREAM_ER/article/details/53485583
您可能感兴趣
- python装饰器初学者教程（Python3.5装饰器原理及应用实例详解）
- python入门知识点总结（深入解析Python小白学习操作列表）
- pythonyield使用场景（Yii框架核心组件类实例详解）
- 怎么用python分析足球（使用Python进行体育竞技分析预测球队成绩）
- python mongodb 基本操作（Python使用pymongo库操作MongoDB数据库的方法实例）
- python数据结构全攻略（详解python数据结构和算法）
- 如何用python识别微信内容（Python 实现微信防撤回功能）
- python端口扫描脚本测试（Python实现的IP端口扫描工具类示例）
- python测试websocket接口（Python如何爬取实时变化的WebSocket数据的方法）
- python中怎么查看函数的用法（Python中函数的基本定义与调用及内置函数详解）
- python 3.10 循环语法（Python3.4学习笔记之常用操作符,条件分支和循环用法示例）
- python写一个二叉树（Python二叉搜索树与双向链表转换算法示例）
- python怎么用代码写出心形（六行python代码的爱心曲线详解）
- python读取文件的方法和区别（浅谈PYTHON 关于文件的操作）
- python中的多线程详解（python多线程抽象编程模型详解）
- python怎样看字符unicode编码（Python3中编码与解码之Unicode与bytes的讲解）
- 苹果自研芯片跑分对比 A16芯片排名靠后，M1系列霸榜（苹果自研芯片跑分对比）
- X86处理器的梦魇 苹果M1自研芯片到底有多强（苹果M1自研芯片到底有多强）
- 泰剧《爱欲之神》Boom kitkong和Great合体杂志（泰剧爱欲之神Boomkitkong和Great合体杂志）
- 素人恋爱综艺火药味十足 男生为赢得芳心集体扯头花，真是出好戏（素人恋爱综艺火药味十足）
- 《囧妈》为何受抵制 春节七部影片撤档背后的责任与博弈（囧妈为何受抵制）
- 提醒 2019年起河南驾考要开设科目五 官方回应来了（2019年起河南驾考要开设科目五）
热门推荐
- 云数据库navicat连不上（Navicat Premium15连接云服务器中的数据库问题及遇到坑）
- php如何发邮件（PHP示例演示发送邮件给某个邮箱）
- nginxdjango部署（详解Django+uwsgi+Nginx上线最佳实战）
- laravel 后台管理框架（关于Laravel-admin的基础用法总结和自定义model详解）
- sql server 数值类型运算（SQL SERVER2012中新增函数之字符串函数CONCAT详解）
- 怎么进入云服务器（云服务器端口怎么打开？）
- pandas数据分组后的处理（详解pandas如何去掉、过滤数据集中的某些值或者某些行？）
- css3定义自定义字体（CSS3字体效果的设置方法小结）
- dedecms列表分页代码（Dedecms文章设置推荐后列表页标题自动加粗的解决方法）
- python 写入d盘文件（python文件写入write的操作）
排行榜
- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9