Macにpyenv + Python + TensorFlowインストールしてmnist使って識別問題

インストール/実行のログ。参考にしたサイトに掲載のコマンドをほぼそのまま実行した時のログです。

参考にしたサイト

環境

macOS Mojave (10.14.1)

実行内容

準備

brew update
brew doctor
sudo mkdir -p /usr/local/include /usr/local/lib /usr/local/sbin
sudo chown -R $(whoami) /usr/local/include /usr/local/lib /usr/local/sbin
xcode-select --install
brew doctor
brew link pcre2

環境変数準備

$ cat .bash_profile 
export PYENV_ROOT=/usr/local/var/pyenv

if which pyenv > /dev/null; then eval "$(pyenv init -)"; fi
$ . .bash_profile
$

python 3.7.1 をインストール ※これはやらなくてよい

3.7.1ではなく、3.6.xをinstallすべきだった。現時点でTensorFlowは3.7には未対応だったので

$ python install 
$ pyenv install 3.7.1
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.1.tar.xz...
-> https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
Installing Python-3.7.1...
python-build: use readline from homebrew
BUILD FAILED (OS X 10.14.1 using python-build 20180424)
Inspect or clean up the working tree at /var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/python-build.20181213215032.11513
Results logged to /var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/python-build.20181213215032.11513.log
Last 10 log lines:
  File "/private/var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/python-build.20181213215032.11513/Python-3.7.1/Lib/ensurepip/__main__.py", line 5, in <module>
    sys.exit(ensurepip._main())
  File "/private/var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/python-build.20181213215032.11513/Python-3.7.1/Lib/ensurepip/__init__.py", line 204, in _main
    default_pip=args.default_pip,
  File "/private/var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/python-build.20181213215032.11513/Python-3.7.1/Lib/ensurepip/__init__.py", line 117, in _bootstrap
    return _run_pip(args + [p[0] for p in _PROJECTS], additional_paths)
  File "/private/var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/python-build.20181213215032.11513/Python-3.7.1/Lib/ensurepip/__init__.py", line 27, in _run_pip
    import pip._internal
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [install] Error 1

エラー対応

zipimport.ZipImportError: can't decompress data; zlib not available という Error が出たら、足りないパッケージがあるのが原因らしい。

$ sudo installer -pkg /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg -target /
Password:
installer: Package name is macOS_SDK_headers_for_macOS_10.14
installer: Installing at base path /
installer: The install was successful.

Python 3.7.1 インストール再実行（成功）

$ pyenv install 3.7.1
python-build: use openssl from homebrew
python-build: use readline from homebrew
Downloading Python-3.7.1.tar.xz...
-> https://www.python.org/ftp/python/3.7.1/Python-3.7.1.tar.xz
Installing Python-3.7.1...
python-build: use readline from homebrew
Installed Python-3.7.1 to /usr/local/var/pyenv/versions/3.7.1

pyenv-virtualenvをインストール

$ brew install pyenv-virtualenv
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
conan           ethereum        kibana          minio-mc        opam
easyengine      exploitdb       minio           node@10
==> Downloading https://github.com/pyenv/pyenv-virtualenv/archive/v1.1.3.tar.gz
==> Downloading from https://codeload.github.com/pyenv/pyenv-virtualenv/tar.gz/v
######################################################################## 100.0%
==> ./install.sh
==> Caveats
To enable auto-activation add to your profile:
  if which pyenv-virtualenv-init > /dev/null; then eval "$(pyenv virtualenv-init -)"; fi
==> Summary
🍺  /usr/local/Cellar/pyenv-virtualenv/1.1.3: 20 files, 62.2KB, built in 8 seconds
$ pyenv rehash
$

$ python -V
Python 2.7.10
$ pyenv virtualenv 3.7.1 TensorFlow
Looking in links: /var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/tmpcigs2r2j
Requirement already satisfied: setuptools in /usr/local/var/pyenv/versions/3.7.1/envs/TensorFlow/lib/python3.7/site-packages (39.0.1)
Requirement already satisfied: pip in /usr/local/var/pyenv/versions/3.7.1/envs/TensorFlow/lib/python3.7/site-packages (10.0.1)
$ pyenv rehash
$ pyenv global TensorFlow
$ python -V
Python 3.7.1
$ pip install --upgrade tensorflow
※エラー発生。

おっと。。。ここで、TensorFlowは現在、python 3.6までしか対応していないことに気づく

python 3.6.7 インストール

仮想環境削除

$ pyenv versions
  system
  3.7.1
  3.7.1/envs/TensorFlow
* TensorFlow (set by /usr/local/var/pyenv/version)
$ pyenv uninstall TensorFlow
pyenv-virtualenv: remove /usr/local/var/pyenv/versions/3.7.1/envs/TensorFlow? y

python 3.6.7インストール

$ pyenv install 3.6.7
$ pyenv versions
pyenv: version `TensorFlow' is not installed (set by /usr/local/var/pyenv/version)
  system
  3.6.7
  3.7.1
$

TensorFlowという名前で仮想環境準備

$ pyenv install 3.6.7
$ python -V
pyenv: version `TensorFlow' is not installed (set by /usr/local/var/pyenv/version)
$ pyenv virtualenv 3.6.7 TensorFlow
Looking in links: /var/folders/nw/gsm23w1x34b0px55kz906jvw0000gn/T/tmpm1px7g0r
Requirement already satisfied: setuptools in /usr/local/var/pyenv/versions/3.6.7/envs/TensorFlow/lib/python3.6/site-packages (39.0.1)
Requirement already satisfied: pip in /usr/local/var/pyenv/versions/3.6.7/envs/TensorFlow/lib/python3.6/site-packages (10.0.1)
mynote:00mnist miki$ 
$ pyenv rehash
$ pyenv global TensorFlow
$ python -V
Python 3.6.7

TensorFlowインストール

$ pip install --upgrade tensorflow
Collecting tensorflow
  Downloading https://files.pythonhosted.org/packages/81/07/be624c7e0a63b080a76b7f6faf417eecdc5f6480f6a740a8bcf8991bce0b/tensorflow-1.12.0-cp36-cp36m-macosx_10_11_x86_64.whl (62.0MB)
    100% |████████████████████████████████| 62.0MB 589kB/s 
Collecting numpy>=1.13.3 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/74/68/2b00ba3c7390354db2a1706291750b6b7e911f6f79c0bd2184ae04f3c6fd/numpy-1.15.4-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (24.5MB)
    100% |████████████████████████████████| 24.5MB 1.4MB/s 
Collecting keras-preprocessing>=1.0.5 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/fc/94/74e0fa783d3fc07e41715973435dd051ca89c550881b3454233c39c73e69/Keras_Preprocessing-1.0.5-py2.py3-none-any.whl
Collecting keras-applications>=1.0.6 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/3f/c4/2ff40221029f7098d58f8d7fb99b97e8100f3293f9856f0fb5834bef100b/Keras_Applications-1.0.6-py2.py3-none-any.whl (44kB)
    100% |████████████████████████████████| 51kB 6.1MB/s 
Collecting gast>=0.2.0 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/5c/78/ff794fcae2ce8aa6323e789d1f8b3b7765f601e7702726f430e814822b96/gast-0.2.0.tar.gz
Collecting protobuf>=3.6.1 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/c7/27/133f225035b9539f2dcfebcdf9a69ff0152f56e0120160ec5c972ea7deb9/protobuf-3.6.1-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (1.2MB)
    100% |████████████████████████████████| 1.2MB 2.9MB/s 
Collecting tensorboard<1.13.0,>=1.12.0 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/e0/d0/65fe48383146199f16dbd5999ef226b87bce63ad5cd73c840cf722637969/tensorboard-1.12.0-py3-none-any.whl (3.0MB)
    100% |████████████████████████████████| 3.1MB 2.9MB/s 
Collecting wheel>=0.26 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/ff/47/1dfa4795e24fd6f93d5d58602dd716c3f101cfd5a77cd9acbe519b44a0a9/wheel-0.32.3-py2.py3-none-any.whl
Collecting grpcio>=1.8.6 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/43/bf/20fc60520e518fe1d1cab864e5210656cd3584d1fbeeeab98075219c3d84/grpcio-1.17.0-cp36-cp36m-macosx_10_7_intel.whl (2.1MB)
    100% |████████████████████████████████| 2.1MB 4.0MB/s 
Collecting absl-py>=0.1.6 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/0c/63/f505d2d4c21db849cf80bad517f0065a30be6b006b0a5637f1b95584a305/absl-py-0.6.1.tar.gz (94kB)
    100% |████████████████████████████████| 102kB 6.2MB/s 
Collecting termcolor>=1.1.0 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/8a/48/a76be51647d0eb9f10e2a4511bf3ffb8cc1e6b14e9e4fab46173aa79f981/termcolor-1.1.0.tar.gz
Collecting astor>=0.6.0 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/35/6b/11530768cac581a12952a2aad00e1526b89d242d0b9f59534ef6e6a1752f/astor-0.7.1-py2.py3-none-any.whl
Collecting six>=1.10.0 (from tensorflow)
  Downloading https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Collecting h5py (from keras-applications>=1.0.6->tensorflow)
  Downloading https://files.pythonhosted.org/packages/44/81/50a0560aac57a33c2a624d9e160735f39d7a6324e3f6f115425a1bf01dd9/h5py-2.8.0-cp36-cp36m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (6.0MB)
    100% |████████████████████████████████| 6.1MB 2.3MB/s 
Requirement not upgraded as not directly required: setuptools in /usr/local/var/pyenv/versions/3.6.7/envs/TensorFlow/lib/python3.6/site-packages (from protobuf>=3.6.1->tensorflow) (39.0.1)
Collecting werkzeug>=0.11.10 (from tensorboard<1.13.0,>=1.12.0->tensorflow)
  Downloading https://files.pythonhosted.org/packages/20/c4/12e3e56473e52375aa29c4764e70d1b8f3efa6682bef8d0aae04fe335243/Werkzeug-0.14.1-py2.py3-none-any.whl (322kB)
    100% |████████████████████████████████| 327kB 9.7MB/s 
Collecting markdown>=2.6.8 (from tensorboard<1.13.0,>=1.12.0->tensorflow)
  Downloading https://files.pythonhosted.org/packages/7a/6b/5600647404ba15545ec37d2f7f58844d690baf2f81f3a60b862e48f29287/Markdown-3.0.1-py2.py3-none-any.whl (89kB)
    100% |████████████████████████████████| 92kB 5.8MB/s 
Installing collected packages: numpy, six, keras-preprocessing, h5py, keras-applications, gast, protobuf, werkzeug, grpcio, markdown, wheel, tensorboard, absl-py, termcolor, astor, tensorflow
  Running setup.py install for gast ... done
  Running setup.py install for absl-py ... done
  Running setup.py install for termcolor ... done
Successfully installed absl-py-0.6.1 astor-0.7.1 gast-0.2.0 grpcio-1.17.0 h5py-2.8.0 keras-applications-1.0.6 keras-preprocessing-1.0.5 markdown-3.0.1 numpy-1.15.4 protobuf-3.6.1 six-1.12.0 tensorboard-1.12.0 tensorflow-1.12.0 termcolor-1.1.0 werkzeug-0.14.1 wheel-0.32.3
You are using pip version 10.0.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
$

mnist を使って画像の数字を識別

mnistは28x28ピクセル・白黒画像・手書き数字のデータセット。
各ピクセルは0(白)~255(黒)の値をとる。6万枚の学習用画像と1万枚のテスト画像。

$ python m.py
Using TensorFlow backend.
60000 train samples
10000 test samples
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_1 (Dense)              (None, 512)               401920    
_________________________________________________________________
dropout_1 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 512)               262656    
_________________________________________________________________
dropout_2 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_3 (Dense)              (None, 10)                5130      
=================================================================
Total params: 669,706
Trainable params: 669,706
Non-trainable params: 0
_________________________________________________________________
Train on 60000 samples, validate on 10000 samples
Epoch 1/20
2018-12-13 23:40:21.075552: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
60000/60000 [==============================] - 3s 49us/step - loss: 0.2455 - acc: 0.9234 - val_loss: 0.1352 - val_acc: 0.9568
Epoch 2/20
60000/60000 [==============================] - 3s 43us/step - loss: 0.1015 - acc: 0.9697 - val_loss: 0.0873 - val_acc: 0.9739
Epoch 3/20
60000/60000 [==============================] - 3s 43us/step - loss: 0.0754 - acc: 0.9779 - val_loss: 0.0755 - val_acc: 0.9792
Epoch 4/20
60000/60000 [==============================] - 3s 44us/step - loss: 0.0603 - acc: 0.9817 - val_loss: 0.0856 - val_acc: 0.9760
Epoch 5/20
60000/60000 [==============================] - 3s 45us/step - loss: 0.0486 - acc: 0.9855 - val_loss: 0.0696 - val_acc: 0.9811
Epoch 6/20
60000/60000 [==============================] - 3s 47us/step - loss: 0.0413 - acc: 0.9874 - val_loss: 0.0925 - val_acc: 0.9786
Epoch 7/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0381 - acc: 0.9887 - val_loss: 0.0758 - val_acc: 0.9821
Epoch 8/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0335 - acc: 0.9902 - val_loss: 0.0728 - val_acc: 0.9825
Epoch 9/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0322 - acc: 0.9910 - val_loss: 0.0831 - val_acc: 0.9818
Epoch 10/20
60000/60000 [==============================] - 3s 48us/step - loss: 0.0280 - acc: 0.9916 - val_loss: 0.0870 - val_acc: 0.9828
Epoch 11/20
60000/60000 [==============================] - 3s 45us/step - loss: 0.0257 - acc: 0.9923 - val_loss: 0.0800 - val_acc: 0.9845
Epoch 12/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0257 - acc: 0.9930 - val_loss: 0.0864 - val_acc: 0.9830
Epoch 13/20
60000/60000 [==============================] - 3s 45us/step - loss: 0.0232 - acc: 0.9932 - val_loss: 0.0972 - val_acc: 0.9828
Epoch 14/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0240 - acc: 0.9933 - val_loss: 0.0925 - val_acc: 0.9839
Epoch 15/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0209 - acc: 0.9943 - val_loss: 0.0966 - val_acc: 0.9841
Epoch 16/20
60000/60000 [==============================] - 3s 47us/step - loss: 0.0226 - acc: 0.9942 - val_loss: 0.0993 - val_acc: 0.9836
Epoch 17/20
60000/60000 [==============================] - 3s 48us/step - loss: 0.0224 - acc: 0.9940 - val_loss: 0.0943 - val_acc: 0.9827
Epoch 18/20
60000/60000 [==============================] - 3s 47us/step - loss: 0.0188 - acc: 0.9950 - val_loss: 0.1026 - val_acc: 0.9841
Epoch 19/20
60000/60000 [==============================] - 3s 46us/step - loss: 0.0185 - acc: 0.9952 - val_loss: 0.1144 - val_acc: 0.9824
Epoch 20/20
60000/60000 [==============================] - 3s 47us/step - loss: 0.0173 - acc: 0.9954 - val_loss: 0.0916 - val_acc: 0.9859
Test loss: 0.09155779591301148
Test accuracy: 0.9859
$

cnn(畳み込みあり）

$ python m.py
Using TensorFlow backend.
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
dropout_1 (Dropout)          (None, 13, 13, 32)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 5408)              0         
_________________________________________________________________
dense_1 (Dense)              (None, 128)               692352    
_________________________________________________________________
dropout_2 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 10)                1290      
=================================================================
Total params: 693,962
Trainable params: 693,962
Non-trainable params: 0
_________________________________________________________________
Train on 60000 samples, validate on 10000 samples
Epoch 1/20
2018-12-15 02:01:25.115542: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
60000/60000 [==============================] - 11s 188us/step - loss: 0.3384 - acc: 0.8985 - val_loss: 0.1039 - val_acc: 0.9681
Epoch 2/20
60000/60000 [==============================] - 11s 175us/step - loss: 0.1366 - acc: 0.9599 - val_loss: 0.0671 - val_acc: 0.9791
Epoch 3/20
60000/60000 [==============================] - 11s 186us/step - loss: 0.1001 - acc: 0.9708 - val_loss: 0.0558 - val_acc: 0.9810
Epoch 4/20
60000/60000 [==============================] - 11s 180us/step - loss: 0.0849 - acc: 0.9750 - val_loss: 0.0479 - val_acc: 0.9833
Epoch 5/20
60000/60000 [==============================] - 11s 175us/step - loss: 0.0758 - acc: 0.9782 - val_loss: 0.0479 - val_acc: 0.9840
Epoch 6/20
60000/60000 [==============================] - 10s 175us/step - loss: 0.0704 - acc: 0.9799 - val_loss: 0.0444 - val_acc: 0.9844
Epoch 7/20
60000/60000 [==============================] - 11s 183us/step - loss: 0.0655 - acc: 0.9808 - val_loss: 0.0432 - val_acc: 0.9860
Epoch 8/20
60000/60000 [==============================] - 12s 192us/step - loss: 0.0664 - acc: 0.9807 - val_loss: 0.0461 - val_acc: 0.9854
Epoch 9/20
60000/60000 [==============================] - 11s 192us/step - loss: 0.0675 - acc: 0.9810 - val_loss: 0.0471 - val_acc: 0.9852
Epoch 10/20
60000/60000 [==============================] - 11s 183us/step - loss: 0.0652 - acc: 0.9816 - val_loss: 0.0458 - val_acc: 0.9860
Epoch 11/20
60000/60000 [==============================] - 11s 184us/step - loss: 0.0654 - acc: 0.9813 - val_loss: 0.0471 - val_acc: 0.9864
Epoch 12/20
60000/60000 [==============================] - 11s 183us/step - loss: 0.0661 - acc: 0.9816 - val_loss: 0.0492 - val_acc: 0.9859
Epoch 13/20
60000/60000 [==============================] - 11s 179us/step - loss: 0.0674 - acc: 0.9820 - val_loss: 0.0457 - val_acc: 0.9868
Epoch 14/20
60000/60000 [==============================] - 11s 177us/step - loss: 0.0701 - acc: 0.9809 - val_loss: 0.0534 - val_acc: 0.9863
Epoch 15/20
60000/60000 [==============================] - 11s 185us/step - loss: 0.0687 - acc: 0.9813 - val_loss: 0.0520 - val_acc: 0.9860
Epoch 16/20
60000/60000 [==============================] - 10s 175us/step - loss: 0.0707 - acc: 0.9809 - val_loss: 0.0481 - val_acc: 0.9863
Epoch 17/20
60000/60000 [==============================] - 11s 180us/step - loss: 0.0733 - acc: 0.9805 - val_loss: 0.0461 - val_acc: 0.9862
Epoch 18/20
60000/60000 [==============================] - 11s 179us/step - loss: 0.0687 - acc: 0.9810 - val_loss: 0.0495 - val_acc: 0.9858
Epoch 19/20
60000/60000 [==============================] - 11s 183us/step - loss: 0.0723 - acc: 0.9809 - val_loss: 0.0473 - val_acc: 0.9858
Epoch 20/20
60000/60000 [==============================] - 11s 179us/step - loss: 0.0738 - acc: 0.9809 - val_loss: 0.0543 - val_acc: 0.9860
Test loss: 0.0542951939858518
Test accuracy: 0.986
$

notebook 001

INPUTとOUTPUTの個人的な備忘録です。