python – California Sunshine

使用Docker+VS Code搭建Go+Python的开发环境

因为项目需要，可能使用Go调用Python 3的接口，趁这个机会实践了一下，如何用Docker+VS Code迅速地搭建开发环境，真的非常非常非常香！

准备Docker Image和Container

先使用Docker迅速的Build出包含了Go，Python 3.7的Image，同时也下载了DataDog/go-python3。这是一个挺不错的Go Module，可以用来调用Python 3的C API，进而调用Python的API（后续写篇文章介绍一下用法）。

FROM debian:10

RUN apt update && \
    apt install -y git golang python3 python3-dev && \
    go get github.com/DataDog/go-python3

FROM debian:10

RUN apt update && \

apt install -y git golang python3 python3-dev && \

go get github.com/DataDog/go-python3

有了Dockerfile，再使用docker build生成Image

docker build -t go_dev:1.0 .

1	docker build -t go_dev:1.0 .

Image生成后，就可以启动Container了，

docker run -it --name go_dev -v H:\go_dev:/go_dev go_dev:1.0

1	docker run -it --name go_dev -v H:\go_dev:/go_dev go_dev:1.0

Container的名字叫go_dev, 同时挂载了H:\go_dev到/go_dev上。Container里的go, python3都已经安装完成。

使用VSCode连接到Docker container

首先需要安装Remote Container的插件，真的是个神器！

安装完成后，可以在左下角一个新的张铎图标，点击后就可以连接刚才启动的Container了。

在菜单中选择，“Attach to Running Container”

在后面的菜单中，选择Container “go_dev”。第一次连接时，需要选择一个目录来写代码了。连接后，打开Terminal也默认调整在Container里的目录。非常方便！

在Remote模式下，代码的很多分析是在Container里完成的，所以在第一次连接完成后，你需要把一些需要的Plugin再安装一下，比如Go语言的支持，Python的支持等等。这个要根据实际的需要。

好了，开发环境搞定，继续板砖写代码吧！

A tricky thing in C++ – Will a member function call by using a raw nullptr crash? Maybe no.

As a Java/Python developer for about 10 years, in my mind, invoking a member function from a raw nullptr, there should have always exceptions, as the following shows.

Maybe you think why I discuss about this naive question. Since recently I found, it’s may not be true for C++ and it depends on how the C++ class defines.

Take a look at the following code written by C++ and execution results.

Surprisingly, the code was executed correctly! It’s tested by VC 2019, GCC 4.8.4/7.3.

I never noticed this but after more time thinking, you may understand the reason. It’s because there is no member field de-referenced in C++ class of “A”.

Let me explain it simply. The function A::hello() will be renamed (there is a full set of naming convention) and compiled into a kind of a C function. And the parameter of pointer of “this” will be added into the function (not that simple, but you can understanding like that). So A::hello() will be compiled to a function like _ZN1A5helloEv(A* this). When you call “a->hello()”, actually, it looks like “_ZN1A5helloEv(a)”. Calling a C function will not cause segment fault, but de-referencing will.

The following code snippet will give you better understanding.

In OOP, this kind of class “A” is not useful. No member fields means a object has no properties. For this case, you should use static function and invoking the function as a class function.

But this tricky behavior is good for us to understand how C++ language works.

Train a simple model to recognize my son

Here is the summary of a personal project – to train a simple deep-learning model and recognize my son’s face.

So far, the hardest part is to prepare the data.

How to Recognize Henry’s Face from Yongqiang Li

Use Atlassian Confluence API to Copy and Create New Pages

For team management, we have a Confluence space to do weekly reporting. In that space, usually, one page is for a week and each page has the same template. So to copy these pages becomes a very boring work. No worries, there are Confluence APIs to help.

Installation and Connection to Confluence

Very simple and pip will help to finish everything.

pip install atlassian-python-api

1	pip install atlassian-python-api

After installation, use the following code to connect to Confluence.

from atlassian import Confluence

confluence = Confluence(
    url='http://localhost:8090',
    username='your_user_name',
    password='your_passwd',
    verify_ssl=False) # in my case, I ignored the SSL verification

from atlassian import Confluence

confluence = Confluence(

url='http://localhost:8090',

username='your_user_name',

password='your_passwd',

verify_ssl=False) # in my case, I ignored the SSL verification

Copy and Create New Page

Before creating new page, you need to get the body content of the template page.

# retrieve the page id.
page_id = confluence.get_page_id("space_a", "template page name")

# retrieve the page, including its content.
page = confluence.get_page_by_id(page_id, expand='body.storage')

# the body HTML.
print(page['body']['storage']['value'])

# retrieve the page id.

page_id = confluence.get_page_id("space_a", "template page name")

# retrieve the page, including its content.

page = confluence.get_page_by_id(page_id, expand='body.storage')

# the body HTML.

print(page['body']['storage']['value'])

Pay attention to the expand=’body.storage’. Without it, the returned page only contains the basic information, no body HTML.

Now you have the body and the rest work is a just for-loop.

for i in range(1, 53):
    print('Creating page for ww{}'.format(i))
    
    # use the same above to retrieve the parent page's id.
    confluence.create_page('', '2019 WW({})'.format(i), content, parent_id=34511303)
    time.sleep(1)

for i in range(1, 53):

print('Creating page for ww{}'.format(i))

# use the same above to retrieve the parent page's id.

confluence.create_page('', '2019 WW({})'.format(i), content, parent_id=34511303)

time.sleep(1)

Also you can use confluence.update_page() to update some existing pages.

References

Create Python Virtual Environment with virtualenv

There are a lot of pros to using a virtual environment to manage different Python environment, which will be independent of each other. The following are the steps to create one under Ubuntu 16.04 with Python 2.7.x

Install the virtualenv with the necessary packages

sudo apt-get install python-pip python-dev python-virtualenv
sudo pip install -U pip

1 2	sudo apt-get install python-pip python-dev python-virtualenv sudo pip install -U pip

Create a folder for the virtualenv

mkdir ~/envfolder/
cd ~/envfolder/
virtualenv --system-site-packages myenv

mkdir ~/envfolder/

cd ~/envfolder/

virtualenv --system-site-packages myenv

Activate & Deactivate the virtual environment

yli@ubuntu:~/envfolder/myenv$ source bin/activate
(myenv) yli@ubuntu:~/envfolder/myenv$

1 2	yli@ubuntu:~/envfolder/myenv$ source bin/activate (myenv) yli@ubuntu:~/envfolder/myenv$

After sourcing the activate script, you can find a “(myenv)” in front of the folder prompt, which is the virtual environment’s name and it has been activated.

Now, the pip can be used to install/update the python modules belongs to this virtual env only.

To deactivate is also very simple.

(myenv) yli@ubuntu:~/envfolder/myenv$ deactivate 
yli@ubuntu:~/envfolder/myenv$

1 2	(myenv) yli@ubuntu:~/envfolder/myenv$ deactivate yli@ubuntu:~/envfolder/myenv$

对Python脚本做简单的profiling

最近事情好多，Blog好久没有更新了。今天上来写写最近解决的一个Python里的性能优化问题。

起因

之前为项目写过一个Sqlite数据库预处理的Python脚本，里面主要做了张新表，把其他表的数据填进去。当时主要考虑到维护性，条理清楚，就没太考虑Performance。之后QA发现模块的运行比原来慢了20倍，因为还是挺快，所以没有当时马上修正。

Profiling

这次Release要修掉这个问题。我的原则是，改进Performance一定要做Profiling，做到有的放矢才。

和Java的VirtualVM类似，Python 2.7也内置了几个Module做Profiling，我选择了cProfile。基本就是如下命令：

python -m cProfile -s tottime python_script

1	python -m cProfile -s tottime python_script

“-s tottime”是让结果用总执行时间排序。

优化之前的执行结果，

        4012898 function calls (4011847 primitive calls) in 139.670 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   281311  131.300    0.000  131.300    0.000 {method 'execute' of 'sqlite3.Cursor' objects}
   121621    4.296    0.000    4.296    0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}
   121618    0.733    0.000    7.110    0.000 a_module.py:510(get_columns)
        1    0.642    0.642  126.630  126.630 a_module.py:32(execute)
   121616    0.470    0.000    7.580    0.000 a_module.py:495(has_field)
    15202    0.440    0.000    7.500    0.000 a_module.py:406(update_parameters)
   364921    0.365    0.000    0.365    0.000 {method 'format' of 'str' objects}
    30405    0.254    0.000    0.254    0.000 {method 'fetchone' of 'sqlite3.Cursor' objects}
        1    0.253    0.253  139.553  139.553 a_module.py:83(preprocess)
  2322518    0.194    0.000    0.194    0.000 {method 'append' of 'list' objects}
...

4012898 function calls (4011847 primitive calls) in 139.670 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)

281311 131.300 0.000 131.300 0.000 {method 'execute' of 'sqlite3.Cursor' objects}

121621 4.296 0.000 4.296 0.000 {method 'fetchall' of 'sqlite3.Cursor' objects}

121618 0.733 0.000 7.110 0.000 a_module.py:510(get_columns)

1 0.642 0.642 126.630 126.630 a_module.py:32(execute)

121616 0.470 0.000 7.580 0.000 a_module.py:495(has_field)

15202 0.440 0.000 7.500 0.000 a_module.py:406(update_parameters)

364921 0.365 0.000 0.365 0.000 {method 'format' of 'str' objects}

30405 0.254 0.000 0.254 0.000 {method 'fetchone' of 'sqlite3.Cursor' objects}

1 0.253 0.253 139.553 139.553 a_module.py:83(preprocess)

2322518 0.194 0.000 0.194 0.000 {method 'append' of 'list' objects}

...

可见罪魁祸首就是sqlite3的Cursor的execute()方法，和原本猜测的也是一样的。优化的手法也很明确就是减少execute()的调用次数，使用batch和合并SQL语句的办法，很容易就用空间换回了时间。

优化之后的结果，

        1048606 function calls (1047555 primitive calls) in 11.653 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
    15235   10.101    0.001   10.101    0.001 {method 'execute' of 'sqlite3.Cursor' objects}
    15202    0.309    0.000    1.448    0.000 a_module.py:422(update_a_parameters)
        1    0.264    0.264   11.557   11.557 a_module.py:88(preprocess)
   250907    0.230    0.000    0.230    0.000 {method 'format' of 'str' objects}
        8    0.226    0.028    0.226    0.028 {method 'fetchall' of 'sqlite3.Cursor' objects}
     7601    0.090    0.000    0.627    0.000 a_module.py:391(update_b_parameters)
   121616    0.075    0.000    1.766    0.000 a_module.py:460(check_and_add_column_if_not_existed)
   167222    0.067    0.000    0.067    0.000 {method 'keys' of 'sqlite3.Row' objects}
...

1048606 function calls (1047555 primitive calls) in 11.653 seconds

Ordered by: internal time

ncalls tottime percall cumtime percall filename:lineno(function)

15235 10.101 0.001 10.101 0.001 {method 'execute' of 'sqlite3.Cursor' objects}

15202 0.309 0.000 1.448 0.000 a_module.py:422(update_a_parameters)

1 0.264 0.264 11.557 11.557 a_module.py:88(preprocess)

250907 0.230 0.000 0.230 0.000 {method 'format' of 'str' objects}

8 0.226 0.028 0.226 0.028 {method 'fetchall' of 'sqlite3.Cursor' objects}

7601 0.090 0.000 0.627 0.000 a_module.py:391(update_b_parameters)

121616 0.075 0.000 1.766 0.000 a_module.py:460(check_and_add_column_if_not_existed)

167222 0.067 0.000 0.067 0.000 {method 'keys' of 'sqlite3.Row' objects}

...

优化之后只是原来的9%的Runtime。

总结

继续坚持用Profiler来做Performance的改进。
边改边用Profiler查看Performance有没有提升。
不要过分优化，否则代码没法看了。😀

References

The Python Profilers

强大的XPath

最近的项目开发中常常要分析XML，深切体会到XPath的强大和方便。Java可以使用DOM4j，Python 2.5以后可以使用了etree。

举个小例子，如果有如下XML：

&lt;A&gt;
   &lt;C&gt;
      &lt;B&gt;1&lt;/B&gt;
   &lt;/C&gt;
   &lt;D&gt;
      &lt;B&gt;2&lt;/B&gt;
   &lt;/D&gt;
&lt;/A&gt;

<A>

<C>

1

</C>

<D>

2

</D>

</A>

要去掉所有的节点，XPath可以如何做呢？

import xml.etree.ELementTree as ET

root = ET.fromstring(xml_str)
b_parent_nodes = root.findall(&quot;.//B/..&quot;) # find parent nodes of all &lt;B&gt; nodes

for i in range(len(b_parent_nodes)):
    node = b_parent_nodes[i].findall(&quot;./B&quot;)[0] # find all &lt;B&gt; nodes under current tag
    b_parent_nodes[i].remove(node)

import xml.etree.ELementTree as ET

root = ET.fromstring(xml_str)

b_parent_nodes = root.findall(".//B/..") # find parent nodes of all nodes

for i in range(len(b_parent_nodes)):

node = b_parent_nodes[i].findall("./B")[0] # find all nodes under current tag

b_parent_nodes[i].remove(node)

XPath的灵活之处就在于，可以用简单明了的Path控制想要读取的节点。

其实还有很多有趣的特性，可以看看XPath Tutorial。

How to migrate ReviewBoard database from sqlite to MySQL

About 3 years ago, I introduced Review Board into our team with a great help from RB group and replaced CodeStrike. We deployed RB on a WinXP VM and the backend database is sqlite. I can’t remember the reason. Maybe I just wanted to save some time. Actually, it has spent a lot of time of our team.

Why? Because sqlite is a lightweight DB and it have bad performace in concurrency scenario. For our team, we have about 20 developers and RB is the most popular daily tools. Now we have to say, we highly depend it to control the quality of team’s code.

Recently, the dababase issue was getting worse. RB pages can’t be created well and there are many warnings/errors on it. And yesterday, the database was dead finally-“The database is locked!”. That’s a message from Django. I googled and the best solution is to use better database to avoid such issues in future.

I took about 3 hours to migrate database to MySQL 5.1. Maybe my experience can help you to save some time.

Step 1. sqlite bump

Go to sqlite.com and download a command line. For window version, you can find it here.

Unzip it and use command to dump sqlite database.

c:\sqlite3.exe reviewboard.db .dump &gt; reviewboard_sqlite_dump

1	c:\sqlite3.exe reviewboard.db .dump > reviewboard_sqlite_dump

reviewboard.db is the database file of sqlite.

Step 2. Convert to MySQL dump

Now we have the sqlite dump file. But you can’t import it into a MySQL database directly. Because some syntax is not supported by MySQL.

I googled and got a free converter to do that. You can find it here and download sqlite3_mysql.zip.

In the deep of the zip , you can find a executable file – sqlite_mysql.exe. Run it and convert the sqlite dump file to a MySQL one.

Step 3. Import MySQL dump

Before that, please create a database in your MySQL database.

mysql -uroot -p reviewboard &lt; mysql_dump

1	mysql -uroot -p reviewboard < mysql_dump

In my case, the dump is about 180M and I waited very long time. (And there may be some warning messages. I don’t know why but for now the migrated server works well. So maybe we can’t ignore them.)

Step 3.1 Alter database structure

Actually, this issue was found when I finished Step 4 and restarted RB. At the beginning, RB worked well. But a minute later, a guy said he could’t submit any comments.

The root cause was the “id” field of some tables lost the property of “AUTO_INCREMENT”. I didn’t know why and SQL in dump file was right. Then I write a very simple .py script to fix this issue.

import os
import string

sql=&quot;alter table %s change id id int not null auto_increment&quot;

tables = &quot;&quot;&quot;
accounts_profile
accounts_profile_starred_groups
accounts_profile_starred_review_requests
accounts_reviewrequestvisit
auth_group
auth_group_permissions
auth_message
auth_permission
auth_user
auth_user_groups
auth_user_user_permissions
changedescs_changedescription
diffviewer_diffset
diffviewer_diffsethistory
diffviewer_filediff
django_admin_log
django_content_type
django_evolution
django_project_version
django_session
django_site
reviews_comment
reviews_defaultreviewer
reviews_defaultreviewer_groups
reviews_defaultreviewer_people
reviews_defaultreviewer_repository
reviews_group
reviews_group_users
reviews_review
reviews_review_comments
reviews_review_screenshot_comments
reviews_reviewrequest
reviews_reviewrequest_changedescs
reviews_reviewrequest_inactive_screenshots
reviews_reviewrequest_screenshots
reviews_reviewrequest_target_groups
reviews_reviewrequest_target_people
reviews_reviewrequestdraft
reviews_reviewrequestdraft_inactive_screenshots
reviews_reviewrequestdraft_screenshots
reviews_reviewrequestdraft_target_groups
reviews_reviewrequestdraft_target_people
reviews_screenshot
reviews_screenshotcomment
scmtools_repository
scmtools_tool
siteconfig_siteconfiguration&quot;&quot;&quot;

if __name__==&#039;__main__&#039;:
    tableNames = tables.splitlines()
    for name in tableNames:
        if name:
            print &#039;Change table %s...&#039; % name
            command = &#039;mysql -uroot -proot reviewboard -e &quot;%s&quot;&#039; % (sql % name)
            print command
            os.system(command)

import os

import string

sql="alter table %s change id id int not null auto_increment"

tables = """

accounts_profile

accounts_profile_starred_groups

accounts_profile_starred_review_requests

accounts_reviewrequestvisit

auth_group

auth_group_permissions

auth_message

auth_permission

auth_user

auth_user_groups

auth_user_user_permissions

changedescs_changedescription

diffviewer_diffset

diffviewer_diffsethistory

diffviewer_filediff

django_admin_log

django_content_type

django_evolution

django_project_version

django_session

django_site

reviews_comment

reviews_defaultreviewer

reviews_defaultreviewer_groups

reviews_defaultreviewer_people

reviews_defaultreviewer_repository

reviews_group

reviews_group_users

reviews_review

reviews_review_comments

reviews_review_screenshot_comments

reviews_reviewrequest

reviews_reviewrequest_changedescs

reviews_reviewrequest_inactive_screenshots

reviews_reviewrequest_screenshots

reviews_reviewrequest_target_groups

reviews_reviewrequest_target_people

reviews_reviewrequestdraft

reviews_reviewrequestdraft_inactive_screenshots

reviews_reviewrequestdraft_screenshots

reviews_reviewrequestdraft_target_groups

reviews_reviewrequestdraft_target_people

reviews_screenshot

reviews_screenshotcomment

scmtools_repository

scmtools_tool

siteconfig_siteconfiguration"""

if __name__=='__main__':

tableNames = tables.splitlines()

for name in tableNames:

if name:

print 'Change table %s...' % name

command = 'mysql -uroot -proot reviewboard -e "%s"' % (sql % name)

print command

os.system(command)

I just went through all tables and alter the property of “id” field if the table has one.

Step 4. Change RB configuration file

Change rb_site_root/conf/settings_local.py to:

DATABASE_ENGINE = &#039;mysql&#039;
DATABASE_NAME = &#039;reviewboard&#039;
DATABASE_USER = &#039;root&#039;
DATABASE_PASSWORD = &#039;xxxxxxxxx&#039;

# comment the former setting
#DATABASE_ENGINE = &#039;sqlite3&#039;
#DATABASE_NAME = &#039;C:\\review-board\\rbsite2\\db\\reviewboard.db&#039;

DATABASE_ENGINE = 'mysql'

DATABASE_NAME = 'reviewboard'

DATABASE_USER = 'root'

DATABASE_PASSWORD = 'xxxxxxxxx'

# comment the former setting

#DATABASE_ENGINE = 'sqlite3'

#DATABASE_NAME = 'C:\\review-board\\rbsite2\\db\\reviewboard.db'

Make sure MySQLdb is installed or there will be error when restart RB.

The best moment came finally! RB restarted successfully and all review request/comment/diff were there. And the performance was improved. You can feel the page is loaded fast. Cheers! 😀

If you have the same problem with me, I hope this post can help and save some time.

Java技术学习之Class文件

Java Class文件是由Java Compiler编译源文件之后产生的。Class文件里保存的就是大名鼎鼎的ByteCode（字节码）。其实在JVM Specification有对它格式的详细描述，我也因此用Python写了一个解析器PyJavap。昨天这个小工具也到了第一个Milestone，支持1.5以前的规范。我心里还是很有成就感的，同时对ByteCode也有了更深了理解，在我的GitHub上可以找到。

现在对Java技术的兴趣越来越深，继续研究吧！ 🙂