Install CKAN 2.5 on Ubuntu 14.04 LTS

Author: Jyun-Yao Huang (Allen; allen501pc@gmail.com)
System Environment: Ubuntu Linux 14.04 LTS (64bits)

1. Update the metadata of packages

$ sudo apt-get update

2. Install apache2, nginx, apache’s module wsgi and the library of postgres- libpq5.

$ sudo apt-get install -y nginx apache2 libapache2-mod-wsgi libpq5

3. Download deb-package of python-ckan.
There are two ways to get CKAN deb package.
First, here you can download my provided deb package in the cloud storage. URL: https://drive.google.com/file/d/0B6PKt2-xNai_QlNfdEZfY05JRlE/view?usp=sharing
Or, download it from CKAN official site.

$ wget http://packaging.ckan.org/python-ckan_2.5-trusty_amd64.deb

4. Install the package.

$ sudo dpkg -i python-ckan_2.5-trusty_amd64.deb

5. Install postgresql.

$ sudo apt-get install postgresql

6. Check the encoding format of Postgresql is UTF-8.

$ sudo -u postgres psql -l

7. Create the database account and the corresponding database. In this case, the account is ckan_default and the database is also called ckan_default.
Please remember your password of your database account.

$ sudo -u postgres createuser -S -D -R -P ckan_default $ sudo -u postgres createdb -O ckan_default ckan_default -E utf-8

8. Edit the file at /etc/ckan/default/production.ini with super user’s permission. Check the settings below:

sqlalchemy.url = postgresql://ckan_default:pass@localhost/ckan_default

Please replace words in bold with the password of your created account for Postgresql.

9. Create the fold for CKAN storage. Type the following command to create storage.

$ sudo mkdir /var/lib/ckan $ chown –R www-data:www-data /var/lib/ckan

10. Edit the file at /etc/ckan/default/production.ini with super user’s permission. Check the settings below:

ckan.storage_path = /var/lib/ckan

11. Download Solr 5.4.1
Warning: Form the CKAN official guide, if you install solr-jetty in Ubuntu 14.04, the JSP of Solr will not be supported. We have to install Solr.
There are two ways to get Solr package.
First, here you can download my provided Solr package in the cloud storage. URL: https://drive.google.com/open?id=0B6PKt2-xNai_Tm9sSnJoZmx1VW8
Or download it from Solr's site.

$ wget http://apache.mirror1.spango.com/lucene/solr/5.4.1/solr-5.4.1.tg

12. Install Solr 5.4.1

$ tar xzf solr-5.4.1.tgz solr-5.4.1/bin/install_solr_service.sh --strip-components=2 $ sudo chmod +x install_solr_service.sh $ sudo ./install_solr_service.sh solr-5.4.1.tgz -f

13. Prepare the CKAN’s schema for Solr
In this case, we prepare a core named ckan for Solr.

$ sudo su root$ mkdir /var/solr/data/ckan root$ touch /var/solr/data/ckan/core.properties root$ cp -a /opt/solr/server/solr/configsets/basic_configs/conf /var/solr/data/ckan root$ mv /var/solr/data/ckan/conf/schema.xml /var/solr/data/ckan1/conf/schema.xml.orig root$ ln -s /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml /var/solr/data/ckan/conf/schema.xml root$ chown -R solr:solr /var/solr root$ exit

14. [Support of Chinese Segmentation] Download the libraries for supporting Chinese segmentation in Solr. There are two packages: mmseg4j-core-x.y.z.jar and mmseg4j-lang-x.y.z.jar to be downloaded. I’ve prepared the zipped file – mmseg4j.zip at https://drive.google.com/open?id=0B6PKt2-xNai_a1luR20wQ0JtenM.

15. [Support of Chinese Segmentation] After downloading mmseg4j.zip and unzipping it into mmseg4j-core-1.10.0.jar and mmseg4j-solr-2.3.0.jar, upload JARs under the folder - /opt/solr/server/solr-webapp/webapp/WEB-INF/lib.

16. [Support of Chinese Segmentation]Modify the schema file at /usr/lib/ckan/default/src/ckan/ckan/config/solr/schema.xml to let Solr support Chinese segmentation.

<fieldType name="text" class="solr.TextField" positionIncrementGap="100"> <analyzer type="index"> <tokenizer class="com.chenlb.mmseg4j.solr.MMSegTokenizerFactory" mode="max-word"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> </analyzer> <analyzer type="query"> <tokenizer class="com.chenlb.mmseg4j.solr.MMSegTokenizerFactory" mode="max-word"/> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt" ignoreCase="true" expand="true"/> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1" generateNumberParts="1" catenateWords="0" catenateNumbers="0" catenateAll="0" splitOnCaseChange="1"/> <filter class="solr.SnowballPorterFilterFactory" language="English" protected="protwords.txt"/> <filter class="solr.LowerCaseFilterFactory"/> <filter class="solr.ASCIIFoldingFilterFactory"/> </analyzer> </fieldType>

17. Restart Solr.

$ sudo service solr restart

18. Serve the following URL to create the core called ckan via your browser.
http://127.0.0.1:8983/solr/admin/cores?action=CREATE&name=ckan&configSet=ckan

19. Edit the file at /etc/ckan/default/production.ini with super user’s permission. Check the setting for Solr below:

solr_url = http://127.0.0.1:8983/solr/ckan

In this case, ckan is the core we used in Solr, so we add ckan as suffix of the URL.

20. Initialize your database. We have to use VirtualEnv which is a virtual environment written in Python. Here, (virtualEnv) is the mark for denoting you are in VirtualEnv.

$ . /usr/lib/ckan/default/bin/activate $ (virtualEnv) cd /usr/lib/ckan/default/src/ckan $ (virtualEnv) paster db init -c /etc/ckan/default/production.ini $ (virtualEnv) deactivate

21. Create Administrator for CKAN.

$ . /usr/lib/ckan/default/bin/activate $ (virtualEnv) cd /usr/lib/ckan/default/src/ckan $ (virtualEnv) paster sysadmin add admin -c /etc/ckan/default/production.ini $ (virtualEnv) deactivate

22. Restart apache2 and nginx.

$ sudo service apache2 restart $ sudo service nginx restart

After that, you will get the following CKAN site.

Support of SSL

With the support of SSL, we can enhance the security of CKAN.

First, we have to apply for an SSL certificate. For freelancers, we can request the SSL certificate from Let’s Encrypt. The method of applying such certificate has been introduced in the official site of Let’s Encrypt. Interested readers can refers to its documents.

After getting an SSL certificate, please following the instructions in the post: Setting up CKAN with SSL.

REFERENCE

1. CKAN, “Installing CKAN from Package,” CKAN. [Online]. Available: http://docs.ckan.org/en/latest/maintaining/installing/install-from-package.html
2. CKAN, “Installing CKAN from Source,” CKAN. [Online]. Available: http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html
3. Koen Vlasinkel, “How to Install Solr 5.2.1 on Ubuntu 14.04,” Digital Ocean Community. [Online]. Available: https://www.digitalocean.com/community/tutorials/how-to-install-solr-5-2-1-on-ubuntu-14-04
4. Hvwaldow, “How to Upgrade to SOLR 5.3.1 on Debian Jessie for Use with CKAN,” CKAN. [Online]. Available: https://gist.github.com/hvwaldow/67fecf80a9790b5c9153#filesolr-upgrade-md
5. CKAN2 安裝教學. URL: http://ckan-docs-tw.readthedocs.io/zh_TW/2.4/install.html
6. CKAN, “Setting up CKAN with SSL,”. [Online]. Available: https://github.com/ckan/ckan/wiki/SSL

[eID] The short survey of electronic ID cards.

[Chinese]
根據新版台灣國民身分證的新聞報導[1]，大部分人會擔心這類措施的相關問題。像是：

1. 為什麼我們政府要遵循國際民航組織ICAO(International Civil Aviation Organization)的標準呢？
2. 為什麼機讀碼這麼像中國政府的台胞證形式呢？

我在此做了一些關於電子身分證(electronic ID, eID)的研究：

ICAO的策略目標強烈地與United Nations Sustainable Development Goals (SDGs)做連結，滿足17項中的13項。包含了確保健康的生命、達到性別平等、建立彈性的基礎設施、確保可承受的消耗與產製樣式、降低國與國之間的不平等性等等。[2]
為了達到這些目標，其中的一個解決方案是推動可機讀旅行文件，稱為ICAO Doc 9303.[3]

為什麼ICAO Doc 9303這麼重要？因為它可讓機器容易讀取你的個人資訊，包含姓名、出生日期、出生地與其他欄位。為了保護這些隱私資料，一系列的安全措施也被採用 (請見[3] 的Part 11與Part 12)。

為了建立eID，德國的新版身分證也採用ICAO Doc 9303 [4]。(事實上，它採用其他方案 ISO/IEC standard 14443 [5]，但也使用ICAO Doc 9303的部分標準[6])

這就是為何我們政府聲稱eID遵循ICAO的國際規範。

另一個問題：「為什麼機讀碼這麼像中國政府的台胞證形式呢？」
可機讀護照(Machine Readable Passport, MRP)採用可機讀區域(Machine Readable Zone)來讓機器輕易讀取[7][8]，這樣的措施也被德國、義大利等國的身分證採用，新式身分證也採用類似的編碼方式。這就是為何新版台胞證有類似的機讀碼。

更多的eID趨勢，可以參見科技新報的報導[9]。

[English]
According to the news of electronic ID card for Taiwanese [1], most of people wonder the problems of this measure. Some problems arises:

• Why does our government follow the standard of ICAO(International Civil Aviation Organization)?
• Why is the machine readable code very similar to that of Taiwanese citizen card which China provided?

I make a short survey on the specification of electronic ID card (abbreviate eID).

ICAO's strategic objective strongly linked to 13 of the 17 United Nations Sustainable Development Goals (SDGs) proposed. Those include ensuring healthy lives, achieving gender equality, building resilient infrastructure, ensuring sustainable consumption and production patterns, reducing inequality among countries and so on.[2]
For achieving these goals, one of the solution is to promote Machine Readable Travel Documents, called ICAO Doc 9303.[3]

Why is the ICAO Doc 9303 important? Because it can let machines easily read your personal information including name, birthday, born town and other fields. For protecting the private data, a series of security measures are adopted. (See Parts 11 and 12 in [3].)

For creating eID cards, ICAO Doc 9303 is one of the solution for German new ID card [4]. (Actually, it adopts another solution called ISO/IEC standard 14443 [5] but also uses part of the specification of ICAO Doc 9303 [6]).

That's why our government say that the new ID card can be compatible with international specification - ICAO.

The other question: "Why is the machine readable code very similar to Taiwanese citizen card which China provided?"

The machine readable passport (MRP) should use a machine readable zone (MRZ) to let machine easily read the information [7][8] which is also adopted by multiple eID cards like German, Italian and so on. The Taiwanese citizen card provided by China use the similar encoding method. That's why the new Taiwanese use the MRZ for machines.
As aforementioned, the security is also ensured.
For further trend of eID, please refer to the post of TechNews.[9]

References

1. 仝澤蓉, "新版身分證樣式曝光 將採中英對照,"  聯合財經網 [Online]. Available:  http://goo.gl/HHKvWx
2. ICAO, "ICAO and the United Nations Sustainable Development Goals," ICAO [Online]. Available:http://www.icao.int/about-icao/aviation-development/Pages/SDG.aspx
3. ICAO, "Machine Readable Travel Documents," ICAO [Online] Available: http://www.icao.int/publications/pages/publication.aspx?docnum=9303
4. Moritz Horsch, Johannes Braun and Alex Wiesmaier, "Mobile eID application for the German identity card," Technische Universität Darmstadt, Technical Report [Online]. Available: https://www.cdc.informatik.tu-darmstadt.de/reports/TR/Mobile_eID_app_for_the_German_ID_card.pdf 
5. "ISO/IEC 14443," Wikipedia [Online]. Available: https://en.wikipedia.org/wiki/ISO/IEC_14443
6. "National identity cards in the European Economic Area," Wikipedia [Online]. Available: https://en.wikipedia.org/wiki/National_identity_cards_in_the_European_Economic_Area
7. "Machine-readable passport," Wikipedia [Online] Available: https://en.wikipedia.org/wiki/Machine-readable_passport#Official_travel_documents 
8. "可機讀護照," Wikiwand [Online] Available:http://goo.gl/iG87tf
9. Sanada Yukimura, "台灣新版晶片身分證 2017 年上路？ 看全球 eID 趨勢," 科技新報 [Online] Available: http://technews.tw/2015/05/11/taiwan-eid-2007/ 

[PHP] Be discreet to use crypt function

Recently, the PHP’s crypt function crashed in my designed new system. In fact, in my database design, the length of password field is 64. It ran fine at CentOS 5.5 with PHP 5.2.x. However, when I moved this system to Debian 7.3.0 with PHP 5.4.x, it was failed to generate the password.

In CentOS 5.5, it used MD5 as default crypt function and the php’s crypt also did (CRYPT_MD5). The MD5 hash generates 34 characters including 12 prefix salt characters like “$1$ … $”and 22 hashed characters in base64 encoding (128 digits / 6 bits_per_characters := 22 characters ), so the legnth of hashed password is enough with the field size = 64.

Unfortunately, the new operating system uses SHA512 crypt function as default crypto scheme and PHP’s crypt function also accepts the crypto scheme as default. In the default setting, the SHA512 hash function generates 102 characters including 16 prefix salt characters like “$6$…$” and 86 hashed characters in based64 encoding (512 digits / 6 bits_per_characters := 86 ). Therefore, it has no extra data length to save whole 102 characters because of the default field size is 64. Finally, the password verifier is also failed. :(

REFERENCE

1. PHP: crypt function.[Recommend]
2. Wiki: MD5
3. Linux Programmer’s Manual : Crypt (3) [Recommend]
4. What is the output length of PHP crypt()?

遠通電收的查詢系統，黑客的聰明選擇

【編按】：目前遠通電收已經把驗證碼機制換成Session-based驗證機制，安全性比較高一些了。
我來幫忙補完好了。
MouseMs 的文章指出，遠通eTag的驗證碼產生方式，是使用Authentication Code + Plaintext 的方式來做驗證。
看樣子沒什麼問題。
但是它最大的敗筆在於三個部分：
1. 把Authentication Code的圖片檔及其相關驗證碼資訊「存到伺服器」 (也就是把輸入的驗證碼存起來的意思)。
    【小實驗】：
    各位可以輸入 https://css.fetc.net.tw/CS/Ajax/GetAuthImage/ohMt1pFQ0is1 ，會得到一個VCode.png1，把這個檔案副檔名改為.png，
    看看是不是都是「XR8DJ」？
2. 把驗證碼的「密文」交給使用者的瀏覽器網頁表單保管(見下圖)。

3. 每次驗證碼產生的「密文」與驗證碼「輸入值」都是一對一對應，產生方式都是固定不變的。
而在伺服器端的驗證碼驗證工作就是：
1. 拿到使用者手動輸入的「驗證碼」以及使用者瀏覽器網頁表單的「密文」
2. 把上面的「驗證碼」與「密文」進行比對；若比對正確，則驗證碼驗證成功。若比對失敗，則驗證碼輸入無效。
但如前所述，它產生的「驗證碼」與「密文」都是一對一對應，無論輸入多少次驗證碼，產生結果都一樣。 亦即無法抵擋「重送攻擊」。
所以黑客只要在驗證碼中輸入「XR8DJ」，密文則填入「ohMt1pFQ0is1」。
按照上面的驗證碼處理工作，就一定破解掉驗證碼。
既然驗證碼可以被破解，黑客的工作就只剩下怎麼破解「車牌號碼」與「車牌證號」了。
提示：用最暴力的做法「字典攻擊法」應該可以破得掉。

SDP, SDN?

Today, I find that CSA(Cloud Security Alliance) have started the SDP(Software Defined Perimeter). The SDP is a framework of security controls to mitigates netowkr-based attacks on Internet-accessible applications by easing conectivities to those until devices/users are authenticated and authorized.

The initial white paper is published here. I take a view of it and find that the developing scheme is like SDN(Software Defined Network).

After some minutes researching…, according to the report, the CSA says that “SDP is being designed to be highly complementary to Software Defined Networks (SDN)”. It seems that SDP is implemeneted for SDN to protect the network attacks.