How to display pictures by ggplot2, RCall and Julia in jupyter? - ggplot2

I tried to use ggplot2 in julia by RCall, like example below
using DataFrames
using RCall
df = DataFrame(randn(10,2),:auto)
#rlibrary ggplot2
#rlibrary ggsci
ggplot(df, aes(x=:x1, y=:x2))+geom_point()
It works well in the REPL, but in jupyter notebook something goes wrong, as the debug info below
REvalError: Warning in (function (filename = "Rplot%03d.png", width = 480, height = 480, :
locale not supported by Xlib: some X ops will operate in C locale
Warning in (function (filename = "Rplot%03d.png", width = 480, height = 480, :
X cannot set locale modifiers
Error in grid.Call(C_stringMetric, as.graphicsAnnot(x$label)) :
X11 font -adobe-helvetica-%s-%s-*-*-%d-*-*-*-*-*-*-*, face 1 at size 9 could not be loaded
Calls: <Anonymous> ... grid.Call -> <Anonymous> -> descentDetails.text -> grid.Call
Stacktrace:
[1] handle_eval_stderr(; as_warning::Bool)
# RCall ~/.julia/packages/RCall/iMDW2/src/io.jl:162
[2] rprint(io::IOContext{IOBuffer}, s::Ptr{VecSxp})
# RCall ~/.julia/packages/RCall/iMDW2/src/io.jl:36
[3] rprint
# ~/.julia/packages/RCall/iMDW2/src/io.jl:46 [inlined]
[4] show
# ~/.julia/packages/RCall/iMDW2/src/io.jl:52 [inlined]
[5] show
# ./multimedia.jl:47 [inlined]
[6] limitstringmime(mime::MIME{Symbol("text/plain")}, x::RObject{VecSxp})
# IJulia ~/.julia/packages/IJulia/e8kqU/src/inline.jl:43
[7] display_mimestring
# ~/.julia/packages/IJulia/e8kqU/src/display.jl:71 [inlined]
[8] display_dict(x::RObject{VecSxp})
# IJulia ~/.julia/packages/IJulia/e8kqU/src/display.jl:102
[9] #invokelatest#2
# ./essentials.jl:708 [inlined]
[10] invokelatest
# ./essentials.jl:706 [inlined]
[11] execute_request(socket::ZMQ.Socket, msg::IJulia.Msg)
# IJulia ~/.julia/packages/IJulia/e8kqU/src/execute_request.jl:112
[12] #invokelatest#2
# ./essentials.jl:708 [inlined]
[13] invokelatest
# ./essentials.jl:706 [inlined]
[14] eventloop(socket::ZMQ.Socket)
# IJulia ~/.julia/packages/IJulia/e8kqU/src/eventloop.jl:8
[15] (::IJulia.var"#15#18")()
# IJulia ./task.jl:411
I tried to save the picture in pdf, png or svg format, and only pdf works well: png and svg will raise error
g = ggplot(df, aes(x=:x1, y=:x2))+geom_point()
ggsave(joinpath(odir,"try.png"),g)
REvalError: Saving 7 x 7 in image
Warning in png_dev(..., res = dpi, units = "in") :
locale not supported by Xlib: some X ops will operate in C locale
Warning in png_dev(..., res = dpi, units = "in") :
X cannot set locale modifiers
Error in grid.Call(C_stringMetric, as.graphicsAnnot(x$label)) :
X11 font -adobe-helvetica-%s-%s-*-*-%d-*-*-*-*-*-*-*, face 1 at size 9 could not be loaded
Calls: <Anonymous> ... grid.Call -> <Anonymous> -> descentDetails.text -> grid.Call
Sup
Run corresponding codes in R works as well.
> library(ggplot2)
> df = data.frame(x1=rnorm(10),x2=rnorm(10))
> df
x1 x2
1 0.18165143 0.38032665
2 -0.62212533 0.52412366
3 0.98072899 0.30052532
4 -0.76721485 -0.63748290
5 0.00894366 0.04508702
6 -0.33166441 0.37056327
7 -0.14156795 -1.06334665
8 -1.08918837 -0.30631348
9 0.22035254 0.78816722
10 0.00356827 -0.62542425
> ggplot(df, aes(x=x1,y=x2))+geom_point()
And it generate pictures here:
Environments
R session
R version 4.1.1 (2021-08-10)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 18.04.3 LTS
Matrix products: default
BLAS: /mnt/Storage/home/yangdongxu/softwares/R/R_4.1_installed/lib/R/lib/libRblas.so
LAPACK: /mnt/Storage/home/yangdongxu/softwares/R/R_4.1_installed/lib/R/lib/libRlapack.so
locale:
[1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=en_US.UTF-8
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] ggplot2_3.3.5
loaded via a namespace (and not attached):
[1] magrittr_2.0.1 tidyselect_1.1.1 munsell_0.5.0 colorspace_2.0-2
[5] R6_2.5.1 rlang_0.4.12 fansi_0.5.0 dplyr_1.0.7
[9] grid_4.1.1 gtable_0.3.0 utf8_1.2.2 DBI_1.1.1
[13] withr_2.4.3 ellipsis_0.3.2 assertthat_0.2.1 digest_0.6.29
[17] tibble_3.1.6 lifecycle_1.0.1 crayon_1.4.2 purrr_0.3.4
[21] farver_2.1.0 vctrs_0.3.8 glue_1.5.1 labeling_0.4.2
[25] compiler_4.1.1 pillar_1.6.4 generics_0.1.1 scales_1.1.1
[29] pkgconfig_2.0.3
Julia: version 1.6.4
So what should I do to display the picture directly in jupyter like python %matplotlib inline?
Thanks.

Related

How to fix UndefVarError issue with #rtransform in Julia?

I am relatively new to Julia, which I started using for data science and machine learning. I have been working with DataFrames.jl and DataFramesMeta.jl which worked just fine until a few days ago.
I noticed that I kept getting the LoadError: UndefVarError: #rtransform not defined when using row-wise operations such as #rtransform and #rsubset, which did not happen previously. Furthermore, when I use the #byrow macro with #transform, I also get the error LoadError: ArgumentError: This path should not be reached.
For some reason, there is no stack trace for the UndefVarError.
The stack trace for the ArgumentError is as follows:
[1] fun_to_vec(ex::Expr; nolhs::Bool, gensym_names::Bool, wrap_byrow::Bool)
# DataFramesMeta ~/.julia/packages/DataFramesMeta/z0mc6/src/parsing.jl:302
[2] (::DataFramesMeta.var"#28#29"{Bool})(ex::Expr)
# DataFramesMeta ./none:0
[3] iterate(::Base.Generator{Vector{Any}, DataFramesMeta.var"#28#29"{Bool}})
# Base ./generator.jl:47
[4] transform!_helper(x::Symbol, args::Expr)
# DataFramesMeta ~/.julia/packages/DataFramesMeta/z0mc6/src/macros.jl:881
[5] var"#transform!"(__source__::LineNumberNode, __module__::Module, x::Any, args::Vararg{Any})
# DataFramesMeta ~/.julia/packages/DataFramesMeta/z0mc6/src/macros.jl:963
[6] eval
# ./boot.jl:373 [inlined]
[7] include_string(mapexpr::typeof(REPL.softscope), mod::Module, code::String, filename::String)
# Base ./loading.jl:1196
[8] invokelatest(::Any, ::Any, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
# Base ./essentials.jl:716
[9] invokelatest(::Any, ::Any, ::Vararg{Any})
# Base ./essentials.jl:714
[10] inlineeval(m::Module, code::String, code_line::Int64, code_column::Int64, file::String; softscope::Bool)
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/eval.jl:238
[11] (::VSCodeServer.var"#66#70"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/eval.jl:162
[12] withpath(f::VSCodeServer.var"#66#70"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams}, path::String)
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/repl.jl:219
[13] (::VSCodeServer.var"#65#69"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/eval.jl:160
[14] hideprompt(f::VSCodeServer.var"#65#69"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/repl.jl:38
[15] (::VSCodeServer.var"#64#68"{Bool, Bool, Bool, Module, String, Int64, Int64, String, VSCodeServer.ReplRunCodeRequestParams})()
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/eval.jl:126
[16] with_logstate(f::Function, logstate::Any)
# Base.CoreLogging ./logging.jl:511
[17] with_logger
# ./logging.jl:623 [inlined]
[18] (::VSCodeServer.var"#63#67"{VSCodeServer.ReplRunCodeRequestParams})()
# VSCodeServer ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/eval.jl:230
[19] #invokelatest#2
# ./essentials.jl:716 [inlined]
[20] invokelatest(::Any)
# Base ./essentials.jl:714
[21] macro expansion
# ~/.vscode/extensions/julialang.language-julia-1.6.28/scripts/packages/VSCodeServer/src/eval.jl:34 [inlined]
[22] (::VSCodeServer.var"#61#62")()
# VSCodeServer ./task.jl:429
I am using the lines using DataFrames and using DataFramesMeta, so I do not have an idea as to why I am running into this error.
Any help is appreciated!

The problem of SQL statement in sqldf in R?

I met some very strainge issue concerning with sqldf.
Example:
sqldf("select REGN, NII from dataframe ")
Even with very simple query this function returns an the following error:
Ошибка в result_create(conn#ptr, statement) :too many SQL variables
Here is the data sample I use as a result of dput(head(dataframe))
structure(list(REGN = c(1L, 21L, 23L, 52L, 53L, 55L), NII = c(5136380L,
30322L, 291812L, 92561L, 120618L, 140652L)), row.names = c(NA,
6L), class = "data.frame")
Edit:
But when I run the similar code to this dataframe:
structure(list(date_start = c("2008-04-01", "2016-04-01", "2019-04-01",
"2020-04-01", "2021-04-01"), date_end = c("2016-04-01", "2019-04-01",
"2020-04-01", "2021-04-01", "3000-01-01")), row.names = c(NA,
-5L), class = c("tbl_df", "tbl", "data.frame"))
everything works.
The results of sessioninfo():
R version 3.5.3 (2019-03-11)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 10 x64 (build 19041)
Matrix products: default
locale:
[1] LC_COLLATE=Russian_Russia.1251
[2] LC_CTYPE=Russian_Russia.1251
[3] LC_MONETARY=Russian_Russia.1251
[4] LC_NUMERIC=C
[5] LC_TIME=Russian_Russia.1251
attached base packages:
[1] stats graphics grDevices utils datasets methods
[7] base
other attached packages:
[1] openxlsx_4.1.0 sqldf_0.4-11
[3] RSQLite_2.1.1 gsubfn_0.7
[5] proto_1.0.0 stringr_1.4.0
[7] reshape2_1.4.3 dplyr_0.8.0.1
[9] readxl_1.3.1 foreign_0.8-71
[11] RevoUtils_11.0.3 RevoUtilsMath_11.0.0
loaded via a namespace (and not attached):
[1] zip_2.0.1 Rcpp_1.0.1 cellranger_1.1.0
[4] pillar_1.3.1 compiler_3.5.3 plyr_1.8.4
[7] tools_3.5.3 RPostgres_1.1.1 digest_0.6.18
[10] bit_1.1-14 memoise_1.1.0 tibble_2.1.1
[13] pkgconfig_2.0.2 rlang_0.3.4 DBI_1.0.0
[16] rstudioapi_0.10 hms_0.4.2 bit64_0.9-7
[19] tidyselect_0.2.5 glue_1.3.1 R6_2.3.0
[22] tcltk_3.5.3 purrr_0.3.2 blob_1.1.1
[25] magrittr_1.5 assertthat_0.2.1 stringi_1.4.3
[28] chron_2.3-53 crayon_1.3.4
The results of ls():
[1] "complete" "con"
[3] "dateAn" "df"
[5] "df_3" "df_4"
[7] "df_inval" "df_inval_q"
[9] "df_itogo" "df_itogo_q"
[11] "df_itogo2" "df_rub"
[13] "df_rub_q" "df1"
[15] "f102_dates" "f102_maket"
[17] "f102_maket_1" "f102_maket_long"
[19] "f102_maket_long_symb" "i"
[21] "j" "k"
[23] "levelFunc" "listdbf"
[25] "m" "MainPath"
[27] "query_compos" "split"
[29] "subtract"
What's the matter?

MatMul in TensorFlow is slower than dot product in numpy

I am observing that on my machine tf.matmul in tensorflow is running significantly slower than dot product in numpy. I have GTX 1080 GPU, and expecting tf.matmul to be at least as fast as when running the code using CPU (numpy).
Environment Info
Operating System
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 16.10
Release: 16.10
Codename: yakkety
Installed version of CUDA and cuDNN:
ls -l /usr/local/cuda-8.0/lib64/libcud*
-rw-r--r-- 1 root root 556000 Feb 22 2017 /usr/local/cuda-8.0/lib64/libcudadevrt.a
lrwxrwxrwx 1 root root 16 Feb 22 2017 /usr/local/cuda-8.0/lib64/libcudart.so -> libcudart.so.8.0
lrwxrwxrwx 1 root root 19 Feb 22 2017 /usr/local/cuda-8.0/lib64/libcudart.so.8.0 -> libcudart.so.8.0.61
-rwxr-xr-x 1 root root 415432 Feb 22 2017 /usr/local/cuda-8.0/lib64/libcudart.so.8.0.61
-rw-r--r-- 1 root root 775162 Feb 22 2017 /usr/local/cuda-8.0/lib64/libcudart_static.a
lrwxrwxrwx 1 voldemaro users 13 Nov 6 2016 /usr/local/cuda-8.0/lib64/libcudnn.so -> libcudnn.so.5
lrwxrwxrwx 1 voldemaro users 18 Nov 6 2016 /usr/local/cuda-8.0/lib64/libcudnn.so.5 -> libcudnn.so.5.1.10
-rwxr-xr-x 1 voldemaro users 84163560 Nov 6 2016 /usr/local/cuda-8.0/lib64/libcudnn.so.5.1.10
-rw-r--r-- 1 voldemaro users 70364814 Nov 6 2016 /usr/local/cuda-8.0/lib64/libcudnn_static.a
TensorFlow Setup
python -c "import tensorflow; print(tensorflow.__version__)"
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcublas.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcudnn.so.5 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcufft.so.8.0 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:135] successfully opened CUDA library libcurand.so.8.0 locally
1.0.0
Code:
'''
Created on Sep 28, 2017
#author: voldemaro
Running on I7/GTX 1080
no MKL
('TF version: ', 'v1.0.0-rc2-15-g47bba63-dirty')
('TF url: ', 'https://github.com/tensorflow/tensorflow/commit/47bba63')
Timing in ms for 2048 x 2048 SVD of type <type 'numpy.float32'> and matmul for 16920 x 2048 of type <type 'numpy.float32'>
numpy default SVD min: 3956.20, median: 4127.75, mean: 4264.41
TF CPU SVD min: 5926.43, median: 5951.70, mean: 5961.43
TF GPU SVD min: 5917.10, median: 6015.87, mean: 6039.63
numpy default .dot product min: 5816.97, median: 5933.43, mean: 5965.22
TF CPU matmul min: 21939.19, median: 22485.99, mean: 22374.69
TF GPU matmul min: 22026.52, median: 22109.97, mean: 22199.43
'''
from scipy import linalg; # for svd
import numpy as np;
import os;
import sys;
import time;
os.environ["TF_CPP_MIN_LOG_LEVEL"]="2" # nospam
import tensorflow as tf;
import gc; gc.disable();
NUM_RUNS = 5;
dtype = np.float32;
N=2048;
M = 16920;
def get_tensorflow_version_url():
import tensorflow as tf
version=tf.__version__
commit = tf.__git_version__
# commit looks like this
# 'v1.0.0-65-g4763edf-dirty'
commit = commit.replace("'","")
if commit.endswith('-dirty'):
dirty = True
commit = commit[:-len('-dirty')]
commit=commit.rsplit('-g', 1)[1]
url = 'https://github.com/tensorflow/tensorflow/commit/'+commit
return url
def get_mkl_version():
import ctypes
import numpy as np
ver = np.zeros(199, dtype=np.uint8)
mkl = ctypes.cdll.LoadLibrary("libmkl_rt.so")
mkl.MKL_Get_Version_String(ver.ctypes.data_as(ctypes.c_char_p), 198)
return ver[ver != 0].tostring()
timeline_counter = 0
run_options = tf.RunOptions(trace_level=tf.RunOptions.FULL_TRACE);
def benchmark(message, func):
time_list = []
for i in range(NUM_RUNS):
start_time = time.time();
func();
time_list.append(time.time()-start_time);
time_list = 1000*np.array(time_list); # get seconds, convert to ms
if len(time_list)>0:
min = np.min(time_list);
median = np.median(time_list);
formatted = ["%.2f"%(d,) for d in time_list[:10]];
result = "min: %8.2f, median: %8.2f, mean: %8.2f"%(min, median, np.mean(time_list))
else:
result = "empty"
print("%-20s %s"%(message, result))
if np.__config__.get_info("lapack_mkl_info"):
print("MKL version", get_mkl_version())
else:
print("no MKL")
print("TF version: ", tf.__git_version__)
print("TF url: ", get_tensorflow_version_url())
svd_array = np.random.random_sample((N,N)).astype(dtype);
another_array = np.random.random_sample((M,N)).astype(dtype);
init_OP = tf.global_variables_initializer();
with tf.device("/gpu:0"):
init_holder_gpu = tf.placeholder(dtype, shape=(M,M));
specVarGPU = tf.random_uniform((N,N), dtype=dtype);
S_gpu = tf.random_uniform((M,N), dtype=dtype);
V_gpu = tf.matmul(tf.matmul(tf.transpose(tf.transpose(tf.conj(S_gpu))), specVarGPU, ), tf.transpose(S_gpu));
[D2_gpu, E1_gpu, E2_gpu] = tf.svd(specVarGPU);
with tf.device("/cpu:0"):
init_holder_cpu = tf.placeholder(dtype, shape=(M,M));
specVarCPU = tf.random_uniform((N,N), dtype=dtype);
S_cpu = tf.random_uniform((M,N), dtype=dtype);
V_cpu = tf.matmul(tf.matmul(tf.transpose(tf.transpose(tf.conj(S_cpu))), specVarCPU, ), tf.transpose(S_cpu));
[D2_cpu, E1_cpu, E2_cpu] = tf.svd(specVarCPU);
V_cpu = tf.matmul(tf.matmul(tf.transpose(tf.transpose(tf.conj(S_cpu))), E1_cpu), tf.transpose(S_cpu));
print("Timing in ms for %d x %d SVD of type %s and matmul for %d x %d of type %s"%(N, N, dtype, M, N, dtype));
def func(): linalg.svd(svd_array)
benchmark("numpy default SVD", func)
config = tf.ConfigProto(allow_soft_placement = True, graph_options=tf.GraphOptions(optimizer_options=tf.OptimizerOptions(opt_level=tf.OptimizerOptions.L0)));
sess = tf.Session(config = config);
sess.run(init_OP);
def func2(): sess.run([D2_cpu.op, E1_cpu.op, E2_cpu.op]);
benchmark("TF CPU SVD", func2);
def func3(): sess.run([D2_gpu.op, E1_gpu.op, E2_gpu.op]);
benchmark("TF GPU SVD", func3);
def func1(): np.transpose(np.asmatrix(another_array)).getH().dot(svd_array).dot(np.transpose(another_array));
benchmark("numpy default .dot product", func1)
def func4(): sess.run([V_cpu]);
benchmark("TF CPU matmul", func4)
def func5(): sess.run([V_gpu])
benchmark("TF GPU matmul", func4)
Apparently tensorflow does not optimize "nested" operations, so
tf.matmul(tf.transpose(tf.conj(a)), x) takes significantly longer time than b = tf.conj(a), c = tf.transpose(b), and d = tf.matmul(c, x).
For SVD, the problem is that there is no GPU Kernel for SVD yet. See here: https://github.com/tensorflow/tensorflow/issues/11588
This means that SVD has to be computed on the CPU, even if the tensors are instantiated on the GPU. For this reason, there's an overhead for transferring data from the GPU to the CPU for computation, then back to the GPU for storing results.
For matmul on the GPU the problem is in the last line of your bechmarking code: you are not calling func5 but func4 again, so you are benchmarking the TF CPU matmul.
Aside from this, there are a few other things you may want to check in your code:
there is no need for the init_holder_cpu and init_holder_gpu vars, as you don't use them
there is no need to run the global_variables_initializer, as there are no variables
you are redefining V_cpu, using one of the outputs from SVD, so you are effectively running both SVD and the matmul in your test
A slightly cleaned up version of the code looks like:
# ... above is the same
print("TF version: ", tf.__git_version__)
print("TF url: ", get_tensorflow_version_url())
svd_array = np.random.random_sample((N,N)).astype(dtype)
another_array = np.random.random_sample((M,N)).astype(dtype)
with tf.device("/gpu:0"):
specVarGPU = tf.random_uniform((N, N), dtype=dtype)
S_gpu = tf.random_uniform((M, N), dtype=dtype)
V_gpu = tf.matmul(tf.matmul(tf.transpose(tf.transpose(tf.conj(S_gpu))), specVarGPU, ), tf.transpose(S_gpu))
D2_gpu, E1_gpu, E2_gpu = tf.svd(specVarGPU)
with tf.device("/cpu:0"):
specVarCPU = tf.random_uniform((N,N), dtype=dtype)
S_cpu = tf.random_uniform((M,N), dtype=dtype)
V_cpu = tf.matmul(tf.matmul(tf.transpose(tf.transpose(tf.conj(S_cpu))), specVarCPU, ), tf.transpose(S_cpu))
D2_cpu, E1_cpu, E2_cpu = tf.svd(specVarCPU)
config = tf.ConfigProto(allow_soft_placement = True, graph_options=tf.GraphOptions(optimizer_options=tf.OptimizerOptions(opt_level=tf.OptimizerOptions.L0)))
def V_numpy():
np.matmul(np.matmul(np.transpose(np.transpose(np.conj(another_array))), svd_array, ), np.transpose(another_array))
with tf.Session(config = config) as sess:
print("Timing in ms for %d x %d SVD of type %s and matmul for %d x %d of type %s"%(N, N, dtype, M, N, dtype))
benchmark("numpy default SVD", lambda: linalg.svd(svd_array))
benchmark("TF CPU SVD", lambda: sess.run([D2_cpu.op, E1_cpu.op, E2_cpu.op]))
benchmark("TF GPU SVD", lambda: sess.run([D2_gpu.op, E1_gpu.op, E2_gpu.op]))
benchmark("numpy MKL matmul", V_numpy)
benchmark("TF CPU matmul", lambda: sess.run([V_cpu.op]))
benchmark("TF GPU matmul", lambda: sess.run([V_gpu.op]))
And outputs (ona an i7 and GTX 1070)
MKL version b'Intel(R) Math Kernel Library Version 2017.0.3 Product Build 20170413 for Intel(R) 64 architecture applications'
TF version: v1.4.0-rc1-11-g130a514
TF url: https://github.com/tensorflow/tensorflow/commit/130a514
Timing in ms for 2048 x 2048 SVD of type <class 'numpy.float32'> and matmul for 16920 x 2048 of type <class 'numpy.float32'>
numpy default SVD min: 3318.42, median: 3320.40, mean: 3320.40
TF CPU SVD min: 4576.71, median: 4577.02, mean: 4577.02
TF GPU SVD min: 14022.59, median: 14172.69, mean: 14172.69
numpy MKL matmul min: 4500.33, median: 4628.01, mean: 4628.01
TF CPU matmul min: 15420.19, median: 15664.84, mean: 15664.84
TF GPU matmul min: 277.80, median: 282.54, mean: 282.54
You can see that the GPU version of matmul is much faster than any CPU implementation, as expected.

can't seem to save chart_Series() as pdf inside a for loop

I can't seem to save chart_Series as pdf using the pdf() and dev.off() inside a for-loop. However, outside the for-loop if I run the code inside the hash box one at a time with incremental value of "i",I am able to produce the graphs. What am I doing wrong?
rm(list = ls(all = TRUE))
setwd("~/Documents/R")
library(tseries)
library(PerformanceAnalytics)
library(quantmod)
library(xts)
library(TTR)
library(Hmisc)
library(Quandl)
Quandl.api_key("yourKeyHere")
startDate = "2012-01-01"
symbols = c(
"ACAD",
"ACM" ,
"AL",
"ALB",
"AMD",
"AMKR",
"AVP",
"BC",
"BID",
"BKD",
"BWA" ,
"BYD" ,
"CAA" ,
"CAR" ,
"CAVM",
"CBL" ,
"CF" ,
"CHK" ,
"CIEN"
)
EOD <- function(symbols, startDate){
for (i in 1:length(symbols)){
stk <-Quandl(paste("EOD/",symbols[i],sep=""), start_date = startDate, type='xts')[,c(8,9,10,11,12)]
sp <-data.frame(Date=index(stk), round(coredata(stk), digits=3))
colnames(sp) <- c("Date","Open", "High", "Low", "Close", "Volume")
setwd("~/Documents/R/STK/csv/eq")
write.csv(sp, paste(paste0(symbols[i]),".csv", sep=""), row.names=FALSE)
}
}
EOD(symbols, startDate)
for (i in 1:length(symbols)){
setwd("~/Documents/R/STK/csv/eq")
stk1 <-getSymbols.csv(symbols[i], src="csv", from = startDate, col.names=c("Open","High","Low","Close","Volume"), auto.assign =FALSE)
colnames(stk1) <- c("Open", "High", "Low", "Close", "Volume")
##########################################################
setwd("~/Documents/R/STK/zs")
name <- paste0(symbols[i])
pdfname <- paste(name, ".pdf", sep="")
pdf(file=pdfname)
chart_Series(stk1 , subset = "2016-01-01::", name = paste0(symbols[i]))
dev.off()
##########################################################
}
######################################################
R version 3.3.2 (2016-10-31)
Platform: x86_64-apple-darwin13.4.0 (64-bit)
Running under: macOS Sierra 10.12.5
locale:
[1] en_GB.UTF-8/en_GB.UTF-8/en_GB.UTF-8/C/en_GB.UTF-8/en_GB.UTF-8
attached base packages:
[1] stats graphics grDevices utils datasets methods base
other attached packages:
[1] KFAS_1.2.8 Quandl_2.9.0 reshape2_1.4.2 Hmisc_4.0-3
[5] ggplot2_2.2.1 Formula_1.2-1 survival_2.41-3 quantstrat_0.9.1739
[9] foreach_1.4.3 blotter_0.9.1741 FinancialInstrument_1.2.0 quantmod_0.4-10
[13] TTR_0.23-1 PerformanceAnalytics_1.4.3541 xts_0.9-7 zoo_1.8-0
[17] tseries_0.10-41 timeSeries_3022.101.2 timeDate_3012.100 lattice_0.20-35
loaded via a namespace (and not attached):
[1] splines_3.3.2 colorspace_1.3-2 htmltools_0.3.6 mgcv_1.8-17 base64enc_0.1-3 rlang_0.1.1 foreign_0.8-69
[8] RColorBrewer_1.1-2 plyr_1.8.4 stringr_1.2.0 munsell_0.4.3 gtable_0.2.0 htmlwidgets_0.8 codetools_0.2-15
[15] labeling_0.3 latticeExtra_0.6-28 knitr_1.16 curl_2.6 htmlTable_1.9 Rcpp_0.12.11 acepack_1.4.1
[22] scales_0.4.1 backports_1.1.0 checkmate_1.8.2 jsonlite_1.5 gridExtra_2.2.1 digest_0.6.12 stringi_1.1.5
[29] grid_3.3.2 quadprog_1.5-5 tools_3.3.2 magrittr_1.5 lazyeval_0.2.0 tibble_1.3.3 cluster_2.0.6
[36] Matrix_1.2-10 data.table_1.10.4 httr_1.2.1 iterators_1.0.8 R6_2.2.2 rpart_4.1-11 nnet_7.3-12
[43] nlme_3.1-131

-1 error message while trying to run H2O XGboost

I am running H2O 3.10.5.3 (on hadoop cdh5.8) on a remote server, connected to my local machine using R (3.4.1) with 'h2o.init(startH2O=F, port=54321)'.
The code below works fine:
Train.hex <- as.h2o(iris)
h2o.gbm(x="Sepal.Length", y="Petal.Width", training_frame = Train.hex)
but xgboost does not work and generates the following error message:
h2o.xgboost(x="Sepal.Length", y="Petal.Width", training_frame = Train.hex)
ERROR: Unexpected HTTP Status code: 500 Server Error (url = http://localhost:54321/3/ModelBuilders/xgboost)
java.lang.ArrayIndexOutOfBoundsException
[1] "java.lang.ArrayIndexOutOfBoundsException: -1"
[2] " hex.ModelBuilder.make(ModelBuilder.java:117)"
[3] " water.api.ModelBuildersHandler.fetch(ModelBuildersHandler.java:35)"
[4] " sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)"
[5] " sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)"
[6] " sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)"
[7] " java.lang.reflect.Method.invoke(Method.java:498)"
[8] " water.api.Handler.handle(Handler.java:63)"
[9] " water.api.RequestServer.serve(RequestServer.java:448)"
[10] " water.api.RequestServer.doGeneric(RequestServer.java:297)"
[11] " water.api.RequestServer.doGet(RequestServer.java:221)"
[12] " javax.servlet.http.HttpServlet.service(HttpServlet.java:707)"
[13] " javax.servlet.http.HttpServlet.service(HttpServlet.java:820)"
[14] " org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:684)"
[15] " org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:503)"
[16] " org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1086)"
[17] " org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:429)"
[18] " org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)"
[19] " org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)"
[20] " org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"
[21] " org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"
[22] " water.JettyHTTPD$LoginHandler.handle(JettyHTTPD.java:183)"
[23] " org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)"
[24] " org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)"
[25] " org.eclipse.jetty.server.Server.handle(Server.java:366)"
[26] " org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:494)"
[27] " org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)"
[28] " org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:971)"
[29] " org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:1033)"
[30] " org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:644)"
[31] " org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)"
[32] " org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)"
[33] " org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)"
[34] " org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)"
[35] " org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)"
[36] " java.lang.Thread.run(Thread.java:745)"
Error in .h2o.doSafeREST(h2oRestApiVersion = h2oRestApiVersion, urlSuffix = page, :
ERROR MESSAGE:
-1
Can anyone help me to understand what is going on?
Many thanks in advance,
Kere
Unfortunately, H2O Xgboost is not supported for the time being on Windows as also the official documentation of H2O states:
Limitations
This section provides a list of XGBoost limitations - some of which
will be addressed in a future release.
...
XGBoost is not supported on Windows.
...
XGBoost is not supported on Windows.
The list of limitations include:
XGBoost is not supported on Windows.
The list of supported platforms includes:Linux or OS X
When I met same issue I open my COLAB account and I try there. It did work for me.