Unable to create Windows agentPool with ACS-Engine - azure-container-service

I am attempting to setup a hybrid ACS-Engine with Kubernetes. I am able to setup the Linux agentPool successfully. When I add the Windows agentPool, I receive an error. I only get this when osType is Windows.
I am using the following to setup the agentPools:
"agentPoolProfiles": [
{
"name": "linuxpool1",
"count": 3,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet",
"osType": "Linux"
},
{
"name": "windowspool2",
"count": 3,
"vmSize": "Standard_D2_v2",
"availabilityProfile": "AvailabilitySet",
"osType": "Windows"
}
],
When the Windows agentPool is included, I receive the following error:
MacBook-Pro:acs-engine appalachios$ ./acs-engine -artifacts zeuscluster zeus_deploy_v1.json
acsengine took 2.42314ms
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x112a551]
goroutine 1 [running]:
github.com/Azure/acs-engine/pkg/api/vlabs.
(*Properties).Validate(0xc420070420, 0x4ce, 0x6ce)
/Users/appalachios/gopath/src/github.com/Azure/acs-
engine/pkg/api/vlabs/validate.go:241 +0x471
github.com/Azure/acs-engine/pkg/api.LoadContainerService(0xc420077500, 0x4ce, 0x6ce,
0xc4200159f0, 0x5, 0x0, 0x0, 0x6ce)
/Users/appalachios/gopath/src/github.com/Azure/acs-engine/pkg/api/apiloader.go:73 +0x398
github.com/Azure/acs-engine/pkg/api.DeserializeContainerService(0xc420077500, 0x4ce, 0x6ce, 0x4ce, 0x6ce, 0x0, 0x0, 0x1271fc0)
/Users/appalachios/gopath/src/github.com/Azure/acs-engine/pkg/api/apiloader.go:30 +0x13a
github.com/Azure/acs-engine/pkg/api.LoadContainerServiceFromFile(0x7fff5fbffbfc, 0x13, 0x0, 0x0, 0x0, 0x0, 0x10b7510)
/Users/appalachios/gopath/src/github.com/Azure/acs-engine/pkg/api/apiloader.go:20 +0x1bd
main.main()
/Users/appalachios/gopath/src/github.com/Azure/acs-engine/acs-engine.go:205 +0x3c2

I am guessing you might be missing windowsProfile in your input API model. E.g.:
"windowsProfile": {
"adminUsername": "azureuser",
"adminPassword": "replacepassword1234$"
}
I will take the feedback to return a better error message in this case. But can you confirm if you have windowsProfile in your API model that you generated the template with?

Related

How to properly use eglfs on Beaglebone Black?

Project description
I am using Yocto to build my embedded Linux for Beaglebone Black. Embedded Linux will be used to host my Qt5 graphicall application.
Embedded Linux image is build for MACHINE defined in poky/meta-yocto-bsp/conf/machine/beaglebone-yocto.conf and not for the one from the repository meta-ti (this one gave me problems).
In the embedded Linux image that I am building I already managed to add (a) Qt5 libraries that my application needs to compile with bitbake and (b) my Qt5 application which sucessfully compiles with bitbake.
Qt5 libraries were appended to the image like this:
IMAGE_INSTALL += "qtbase qwt-qt5"
And my Qt application was appended like this:
IMAGE_INSTALL += "application"
So libraries are installed through recipes qtbase and qwt-qt5 while my application was installed through recipe application which I wrote myself.
I can post a list of files that each recipe installs on the target's root file system:
┌───┐
│ $ │ ziga > ziga--hypervisor > build--beaglbone_black_c2
└─┬─┘ /dev/pts/1
└─> oe-pkgdata-util list-pkg-files qtbase qwt-qt5 application
application:
/usr/bin/executable
qtbase:
/usr/lib/libQt5Concurrent.so.5
/usr/lib/libQt5Concurrent.so.5.14
/usr/lib/libQt5Concurrent.so.5.14.2
/usr/lib/libQt5Core.so.5
/usr/lib/libQt5Core.so.5.14
/usr/lib/libQt5Core.so.5.14.2
/usr/lib/libQt5DBus.so.5
/usr/lib/libQt5DBus.so.5.14
/usr/lib/libQt5DBus.so.5.14.2
/usr/lib/libQt5Gui.so.5
/usr/lib/libQt5Gui.so.5.14
/usr/lib/libQt5Gui.so.5.14.2
/usr/lib/libQt5Network.so.5
/usr/lib/libQt5Gui.so.5.14
/usr/lib/libQt5Gui.so.5.14.2
/usr/lib/libQt5Network.so.5
/usr/lib/libQt5Network.so.5.14
/usr/lib/libQt5Network.so.5.14.2
/usr/lib/libQt5OpenGL.so.5
/usr/lib/libQt5OpenGL.so.5.14
/usr/lib/libQt5OpenGL.so.5.14.2
/usr/lib/libQt5PrintSupport.so.5
/usr/lib/libQt5PrintSupport.so.5.14
/usr/lib/libQt5PrintSupport.so.5.14.2
/usr/lib/libQt5Sql.so.5
/usr/lib/libQt5Sql.so.5.14
/usr/lib/libQt5Sql.so.5.14.2
/usr/lib/libQt5Test.so.5
/usr/lib/libQt5Test.so.5.14
/usr/lib/libQt5Test.so.5.14.2
/usr/lib/libQt5Widgets.so.5
/usr/lib/libQt5Widgets.so.5.14
/usr/lib/libQt5Widgets.so.5.14.2
/usr/lib/libQt5XcbQpa.so.5
/usr/lib/libQt5XcbQpa.so.5.14
/usr/lib/libQt5XcbQpa.so.5.14.2
/usr/lib/libQt5Xml.so.5
/usr/lib/libQt5Xml.so.5.14
/usr/lib/libQt5Xml.so.5.14.2
qwt-qt5:
/usr/lib/libqwt.so.6
/usr/lib/libqwt.so.6.1
/usr/lib/libqwt.so.6.1.4
Application can't display graphics
Now I login to the target and run my application which can not display the graphics:
root#beaglebone-yocto:~# executable
2020/09/20 10:44:26.560: Warning: Could not connect to display
2020/09/20 10:44:26.581: Info: Could not load the Qt platform plugin "xcb" in "" even though it was found.
2020/09/20 10:44:26.581: Fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix the problem.
Available platform plugins are: minimal, offscreen, vnc, xcb.
Aborted
It looks like I have to somehow configure "Qt Platform Abstraction" (QPA) which is the platform abstraction layer for Qt5 with a lot of plugins (source) that also include a lot of platforms.
My application only uses a single window so I could probably go with qeglfs or qlinuxfb. But because Beaglebone Black integrates Imagination's GPU PowerVR SGX530 which supports OpenGL|ES 2.0 (source) and this is why I want to go with qeglfs platform. Note however that currently qeglfs platform was not listed in the above error report:
Available platform plugins are: minimal, offscreen, vnc, xcb.
I managed to install the qeglfs platform by creating a qtbase_%.bbappend file with this content:
PACKAGECONFIG:append = " eglfs gles2"
Note that this activates qtbase recipe's lines:
PACKAGECONFIG[eglfs] = "-eglfs,-noeglfs,drm"
PACKAGECONFIG[gles2] = "-opengl es2,,virtual/libgles2 virtual/egl"
These lines make sure that compilation includes eglfs as well as OpenGL|ES i.e. gles2. I chose to also include gles2 after reading / watching this (source, source):
So I asumed that eglfs also needs gles2.
Now eglfs is installed and is listed when I try to execute my application in the same way as before:
root#beaglebone-yocto:~# executable
2020/09/20 10:44:26.560: Warning: Could not connect to display
2020/09/20 10:44:26.581: Info: Could not load the Qt platform plugin "xcb" in "" even though it was found.
2020/09/20 10:44:26.581: Fatal: This application failed to start because no Qt platform plugin could be initialized. Reinstalling the application may fix the problem.
Available platform plugins are: eglfs, minimal, offscreen, vnc, xcb.
Aborted
But when I try to use eglfs I get errors:
root#beaglebone-yocto:~# env QT_QPA_PLATFORM=eglfs executable
2020/09/20 11:01:17.525: Warning: QStandardPaths: XDG_RUNTIME_DIR not set, defaulting to '/tmp/runtime-root'
error: XDG_RUNTIME_DIR not set in the environment
error: XDG_RUNTIME_DIR not set in the environment
error: XDG_RUNTIME_DIR not set in the environment
2020/09/20 11:01:17.533: Fatal: Could not initialize egl display
Aborted
All the errors except for the last one can be fixed by adding some enviromental variables like this:
root#beaglebone-yocto:~# env QT_QPA_PLATFORM=eglfs XDG_RUNTIME_DIR=/tmp/runtime-root executable
2020/09/20 11:01:17.533: Fatal: Could not initialize egl display
Aborted
But as you can see the last error remains and I don't know how to fix it.
Debug info reveals nothing
I tried exporting some enviromental variables to make debug more verbose and then I ran application again:
root#beaglebone-yocto:~# export QT_LOGGING_RULES=true
root#beaglebone-yocto:~# export QSG_INFO=1
root#beaglebone-yocto:~# export QT_QPA_EGLFS_DEBUG=1
root#beaglebone-yocto:~# export QT_DEBUG_PLUGINS=1
root#beaglebone-yocto:~# export QT_QPA_PLATFORM=eglfs
root#beaglebone-yocto:~# export QT_QPA_EGLFS_INTEGRATION=none
root#beaglebone-yocto:~# export XDG_RUNTIME_DIR=/tmp/runtime-root
root#beaglebone-yocto:~# executable
2022/02/17 20:54:28.448: Debug: QFactoryLoader::QFactoryLoader() checking directory path "/usr/lib/plugins/platforms" ...
2022/02/17 20:54:28.450: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqeglfs.so"
2022/02/17 20:54:28.451: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqeglfs.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"eglfs"
]
},
"archreq": 0,
"className": "QEglFSIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.451: Debug: Got keys from plugin meta data ("eglfs")
2022/02/17 20:54:28.451: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqlinuxfb.so"
2022/02/17 20:54:28.452: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqlinuxfb.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"linuxfb"
]
},
"archreq": 0,
"className": "QLinuxFbIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.453: Debug: Got keys from plugin meta data ("linuxfb")
2022/02/17 20:54:28.453: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqminimal.so"
2022/02/17 20:54:28.454: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqminimal.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"minimal"
]
},
"archreq": 0,
"className": "QMinimalIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.454: Debug: Got keys from plugin meta data ("minimal")
2022/02/17 20:54:28.454: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqminimalegl.so"
2022/02/17 20:54:28.455: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqminimalegl.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"minimalegl"
]
},
"archreq": 0,
"className": "QMinimalEglIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.456: Debug: Got keys from plugin meta data ("minimalegl")
2022/02/17 20:54:28.456: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqoffscreen.so"
2022/02/17 20:54:28.457: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqoffscreen.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"offscreen"
]
},
"archreq": 0,
"className": "QOffscreenIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.457: Debug: Got keys from plugin meta data ("offscreen")
2022/02/17 20:54:28.457: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqvnc.so"
2022/02/17 20:54:28.458: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqvnc.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"vnc"
]
},
"archreq": 0,
"className": "QVncIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.458: Debug: Got keys from plugin meta data ("vnc")
2022/02/17 20:54:28.459: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/platforms/libqxcb.so"
2022/02/17 20:54:28.459: Warning: Found metadata in lib /usr/lib/plugins/platforms/libqxcb.so, metadata=
{
"IID": "org.qt-project.Qt.QPA.QPlatformIntegrationFactoryInterface.5.3",
"MetaData": {
"Keys": [
"xcb"
]
},
"archreq": 0,
"className": "QXcbIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.460: Debug: Got keys from plugin meta data ("xcb")
2022/02/17 20:54:28.460: Debug: QFactoryLoader::QFactoryLoader() checking directory path "/usr/bin/platforms" ...
2022/02/17 20:54:28.512: Debug: loaded library "/usr/lib/plugins/platforms/libqeglfs.so"
2022/02/17 20:54:28.513: Debug: QFactoryLoader::QFactoryLoader() checking directory path "/usr/lib/plugins/egldeviceintegrations" ...
2022/02/17 20:54:28.514: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/egldeviceintegrations/libqeglfs-emu-integration.so"
2022/02/17 20:54:28.515: Warning: Found metadata in lib /usr/lib/plugins/egldeviceintegrations/libqeglfs-emu-integration.so, metadata=
{
"IID": "org.qt-project.qt.qpa.egl.QEglFSDeviceIntegrationFactoryInterface.5.5",
"MetaData": {
"Keys": [
"eglfs_emu"
]
},
"archreq": 0,
"className": "QEglFSEmulatorIntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.516: Debug: Got keys from plugin meta data ("eglfs_emu")
2022/02/17 20:54:28.516: Debug: QFactoryLoader::QFactoryLoader() looking at "/usr/lib/plugins/egldeviceintegrations/libqeglfs-x11-integration.so"
2022/02/17 20:54:28.517: Warning: Found metadata in lib /usr/lib/plugins/egldeviceintegrations/libqeglfs-x11-integration.so, metadata=
{
"IID": "org.qt-project.qt.qpa.egl.QEglFSDeviceIntegrationFactoryInterface.5.5",
"MetaData": {
"Keys": [
"eglfs_x11"
]
},
"archreq": 0,
"className": "QEglFSX11IntegrationPlugin",
"debug": false,
"version": 331264
}
2022/02/17 20:54:28.517: Debug: Got keys from plugin meta data ("eglfs_x11")
2022/02/17 20:54:28.517: Debug: QFactoryLoader::QFactoryLoader() checking directory path "/usr/bin/egldeviceintegrations" ...
2022/02/17 20:54:28.543: Fatal: Could not initialize egl display
Now I get much more data, but not regarding the last error! That one remains unverbose as before. Really useless actually!
GPU is discovered by kernel - but not used
I also made sure that I added status = "okay" inside the devicetree's GPU device node. This enabled the GPU probing. So now I can see it in dmesg:
[ 0.000000] AM335X ES2.1 (sgx neon)
I also installed executable kmscube which I could sucessfully execute and it rendered a cube with shaders. But it was slow and top revealed that it used 100% of my CPU! So probably GPU wasn't used in this case!
My display controllers & panels
I have two connectors on my device - first is HDMI and the second is my embedded panel. I installed modetest DRM & KVM utility which shows, how display controllers and panels are set up:
trying to open device 'i915'...failed
trying to open device 'amdgpu'...failed
trying to open device 'radeon'...failed
trying to open device 'nouveau'...failed
trying to open device 'vmwgfx'...failed
trying to open device 'omapdrm'...failed
trying to open device 'exynos'...failed
trying to open device 'tilcdc'...done
Encoders:
id crtc type possible crtcs possible clones
33 32 LVDS 0x00000001 0x00000001
35 0 TMDS 0x00000001 0x00000002
Connectors:
id encoder status name size (mm) modes encoders
34 33 connected LVDS-1 0x0 1 33
modes:
index name refresh (Hz) hdisp hss hse htot vdisp vss vse vtot)
#0 800x480 34.89 800 840 888 928 480 493 496 525 17000 flags: nhsync, nvsync; type: preferred, driver
props:
1 EDID:
flags: immutable blob
blobs:
value:
2 DPMS:
flags: enum
enums: On=0 Standby=1 Suspend=2 Off=3
value: 0
5 link-status:
flags: enum
enums: Good=0 Bad=1
value: 0
6 non-desktop:
flags: immutable range
values: 0 1
value: 0
4 TILE:
flags: immutable blob
blobs:
value:
20 CRTC_ID:
flags: object
value: 32
36 0 disconnected HDMI-A-1 0x0 0 35
props:
1 EDID:
flags: immutable blob
blobs:
value:
2 DPMS:
flags: enum
enums: On=0 Standby=1 Suspend=2 Off=3
value: 0
5 link-status:
flags: enum
enums: Good=0 Bad=1
value: 0
6 non-desktop:
flags: immutable range
values: 0 1
value: 0
4 TILE:
flags: immutable blob
blobs:
value:
20 CRTC_ID:
flags: object
value: 0
CRTCs:
id fb pos size
32 37 (0,0) (800x480)
#0 800x480 34.89 800 840 888 928 480 493 496 525 17000 flags: nhsync, nvsync; type: preferred, driver
props:
22 ACTIVE:
flags: range
values: 0 1
value: 1
23 MODE_ID:
flags: blob
blobs:
value:
68420000200348037803a0030000e001
ed01f0010d020000230000000a000000
48000000383030783438300000000000
00000000000000000000000000000000
00000000
19 OUT_FENCE_PTR:
flags: range
values: 0 18446744073709551615
value: 0
24 VRR_ENABLED:
flags: range
values: 0 1
value: 0
Planes:
id crtc fb CRTC x,y x,y gamma size possible crtcs
31 32 37 0,0 0,0 0 0x00000001
formats: BG16 RG24 XR24
props:
8 type:
flags: immutable enum
enums: Overlay=0 Primary=1 Cursor=2
value: 1
17 FB_ID:
flags: object
value: 37
18 IN_FENCE_FD:
flags: signed range
values: -1 2147483647
value: -1
20 CRTC_ID:
flags: object
value: 32
13 CRTC_X:
flags: signed range
values: -2147483648 2147483647
value: 0
14 CRTC_Y:
flags: signed range
values: -2147483648 2147483647
value: 0
15 CRTC_W:
flags: range
values: 0 2147483647
value: 800
16 CRTC_H:
flags: range
values: 0 2147483647
value: 480
9 SRC_X:
flags: range
values: 0 4294967295
value: 0
10 SRC_Y:
flags: range
values: 0 4294967295
value: 0
11 SRC_W:
flags: range
values: 0 4294967295
value: 52428800
12 SRC_H:
flags: range
values: 0 4294967295
value: 31457280
Frame buffers:
id size pitch
Here everything looks correct and in general my displays are working fine. So this probably isn't root of my problem.
This looks like a similar issue on the Beagleboard forum. The dev was able to fix it doing this:
"I updated the kernel using these options: ./update_kernel.sh --ti-channel --lts-4_14 --sgx
I had to install an additional package: apt install libdrm-dev
I had to do a custom GBM library install, outlined here 9. Seems not everything was set up correctly as part of the image."
Source

In Remix and Ropsten, the bytecode of a contract is different

The contract which I compile in Remix and depoly on Ropsten.
pragma solidity ^0.5.11;
contract Deployer {
constructor() public {
bytes memory bytecode = hex'61000061000061000061000061006161000301610000619789f100';
assembly {
return (add(bytecode, 0x20), mload(bytecode))
}
}
}
Its bytecode in Remix is
{
"linkReferences": {},
"object": "6080604052348015600f57600080fd5b5060606040518060400160405280601b81526020017f61000061000061000061000061006161000301610000619789f10000000000008152509050805160208201f3fe",
"opcodes": "PUSH1 0x80 PUSH1 0x40 MSTORE CALLVALUE DUP1 ISZERO PUSH1 0xF JUMPI PUSH1 0x0 DUP1 REVERT JUMPDEST POP PUSH1 0x60 PUSH1 0x40 MLOAD DUP1 PUSH1 0x40 ADD PUSH1 0x40 MSTORE DUP1 PUSH1 0x1B DUP2 MSTORE PUSH1 0x20 ADD PUSH32 0x61000061000061000061000061006161000301610000619789F1000000000000 DUP2 MSTORE POP SWAP1 POP DUP1 MLOAD PUSH1 0x20 DUP3 ADD RETURN INVALID ",
"sourceMap": "28:241:0:-;;;53:213;8:9:-1;5:2;;;30:1;27;20:12;5:2;53:213:0;85:21;:83;;;;;;;;;;;;;;;;;;;238:8;232:15;225:4;215:8;211:19;203:45"
}
Its runtime bytecode is
{
"linkReferences": {},
"object": "6080604052600080fdfea265627a7a7231582046db7afc80e31555b0d8a4dd3357ac5c36930b00fef8a32c2746b725244550ae64736f6c63430005110032",
"opcodes": "PUSH1 0x80 PUSH1 0x40 MSTORE PUSH1 0x0 DUP1 REVERT INVALID LOG2 PUSH6 0x627A7A723158 KECCAK256 CHAINID 0xDB PUSH27 0xFC80E31555B0D8A4DD3357AC5C36930B00FEF8A32C2746B7252445 POP 0xAE PUSH5 0x736F6C6343 STOP SDIV GT STOP ORIGIN ",
"sourceMap": "28:241:0:-;;;;;"
}
But when I deploy it to the Ropsten:https://ropsten.etherscan.io/address/0xCE1e482Bb5600f7DE9d316bcd30fb53cBAd4DcBe#code
Its bytecode is
0x61000061000061000061000061006161000301610000619789f100
I want to know what is the reason for this situation.
The Remix bytecode export contains the bytecode variable definition and assembly block without the constructor being executed - just the actual definitions compiled to bytecode.
Constructor usually returns the contract instance (including its parent contracts and imported libraries). But in this case, the assembly block forces the EVM to not return the contract instance, but to return the value of the bytecode variable from the constructor.
Then the deployment process continues as usual - the EVM stores the constructor return value (usually the contract instance, now the bytecode) into the contract storage.

Databricks spark_jar_task failed when submitted via API

I am using to submit a sample spark_jar_task
My sample spark_jar_task request to calculate Pi :
"libraries": [
{
"jar": "dbfs:/mnt/test-prd-foundational-projects1/spark-examples_2.11-2.4.5.jar"
}
],
"spark_jar_task": {
"main_class_name": "org.apache.spark.examples.SparkPi"
}
Databricks sysout logs where it prints the Pi value as expected
....
(This session will block until Rserve is shut down) Spark package found in SPARK_HOME: /databricks/spark DATABRICKS_STDOUT_END-19fc0fbc-b643-4801-b87c-9d22b9e01cd2-1589148096455
Executing command, time = 1589148103046.
Executing command, time = 1589148115170.
Pi is roughly 3.1370956854784273
Heap
.....
Spark_jar_task though prints the PI value in log, the job got terminated with failed status without stating the error. Below are response of api /api/2.0/jobs/runs/list/?job_id=23.
{
"runs": [
{
"job_id": 23,
"run_id": 23,
"number_in_job": 1,
"state": {
"life_cycle_state": "TERMINATED",
"result_state": "FAILED",
"state_message": ""
},
"task": {
"spark_jar_task": {
"jar_uri": "",
"main_class_name": "org.apache.spark.examples.SparkPi",
"run_as_repl": true
}
},
"cluster_spec": {
"new_cluster": {
"spark_version": "6.4.x-scala2.11",
......
.......
Why the job failed here? Any suggestions will be appreciated!
EDIT :
The errorlog says
20/05/11 18:24:15 INFO ProgressReporter$: Removed result fetcher for 740457789401555410_9000204515761834296_job-34-run-1-action-34
20/05/11 18:24:15 WARN ScalaDriverWrapper: Spark is detected to be down after running a command
20/05/11 18:24:15 WARN ScalaDriverWrapper: Fatal exception (spark down) in ReplId-a46a2-6fb47-361d2
com.databricks.backend.common.rpc.SparkStoppedException: Spark down:
at com.databricks.backend.daemon.driver.DriverWrapper.getCommandOutputAndError(DriverWrapper.scala:493)
at com.databricks.backend.daemon.driver.DriverWrapper.executeCommand(DriverWrapper.scala:597)
at com.databricks.backend.daemon.driver.DriverWrapper.runInnerLoop(DriverWrapper.scala:390)
at com.databricks.backend.daemon.driver.DriverWrapper.runInner(DriverWrapper.scala:337)
at com.databricks.backend.daemon.driver.DriverWrapper.run(DriverWrapper.scala:219)
at java.lang.Thread.run(Thread.java:748)
20/05/11 18:24:17 INFO ShutdownHookManager: Shutdown hook called
I found answer from this post https://github.com/dotnet/spark/issues/126
Looks like, we shouldnt deliberately call
spark.stop()
when running as a jar in databricks

SSL Error: Handshake failed with fatal error - Querying fabric-sdk-rest server on a Fabric Network with TLS enabled

I started a Multi-Host Fabric Network usind docker swarm made up of 1 CA-server, 1 Orderer, 2 Peers (both in Org1, one on PC1 and one on PC2), 2 CouchBD (one for each Peer) with fabric-sdk-rest running on PC2.
Now if I disable TLS in the Fabric Network, everything works fine. But if i enable the TLS in the network, the SDK cannot connect to the peers failing to query.
Here I show the configuration of the network and the fabric-sdk-rest:
(crypto-config.yaml)
OrdererOrgs:
- Name: Orderer
Domain: example.com
Specs:
- Hostname: orderer
PeerOrgs:
- Name: Org1
Domain: org1.example.com
Template:
Count: 2
Users:
Count: 0
(datasources.json)
{
"db": {
"name": "db",
"connector": "memory"
},
"fabricDataSource": {
"name": "fabricDataSource",
"connector": "fabric",
"keyStoreFile": "/tmp/fabricSDKStore",
"fabricUser": {
"username": "Admin#org1.example.com",
"mspid": "Org1MSP",
"cryptoContent": {
"privateKey":"$HOME/mynetwork/crypto-config/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp/keystore/KEY_sk",
"signedCert":"$HOME/mynetwork/crypto-config/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp/signcerts/Admin#org1.example.com-cert.pem"
}
},
"COMMENT_orgs":"Referenced by peers to avoid having to configure the same file location multiple times. Change CACertFile locations for your fabric",
"orgs": [
{ "name":"org1", "CACertFile":"$HOME/mynetwork/crypto-config/peerOrganizations/org1.example.com/ca/ca.org1.example.com-cert.pem"}
],
"COMMENT_peers" : "Configured array is for use with the fabric-sample when running it in a local docker set up. eventURL and publicCertFile not currently used.",
"peers": [
{ "requestURL":"grpcs://peer1.org1.example.com:7051", "eventURL":"grpcs://peer1.org1.example.com:7053", "orgIndex":"0", "publicCertFile":"$HOME/mynetwork/crypto-config/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/msp/signcerts/peer1.org1.example.com-cert.pem", "hostname":"peer1" }
],
"COMMENT_peers_secure" : "UNUSED. This is a copy of the above with grpcs URLs. Replace peers content with this if grpcs urls are needed.",
"peers-secure": [
{ "requestURL":"grpcs://peer1.org1.example.com:7051", "eventURL":"grpcs://peer1.org1.example.com:7053", "orgIndex":"0", "publicCertFile":"$HOME/mynetwork/crypto-config/peerOrganizations/org1.example.com/peers/peer1.org1.example.com/msp/signcerts/peer1.org1.example.com-cert.pem", "hostname":"peer1" }
],
"orderers": [
{ "url":"grpcs://orderer.example.com:7050", "CACertFile":"$HOME/mynetwork/crypto-config/ordererOrganizations/example.com/ca/ca.example.com-cert.pem", "publicCertFile": "$HOME/mynetwork/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp/signcerts/orderer.example.com-cert.pem", "hostname":"orderer"}
],
"COMMENT_orderers_secure" : "UNUSED. This is a copy of the above with grpcs URLs. Replace orderers content with this if grpcs urls are needed.",
"orderers-secure": [
{ "url":"grpcs://orderer.example.com:7050", "CACertFile":"$HOME/mynetwork/crypto-config/ordererOrganizations/example.com/ca/ca.example.com-cert.pem", "publicCertFile": "$HOME/mynetwork/crypto-config/ordererOrganizations/example.com/orderers/orderer.example.com/msp/signcerts/orderer.example.com-cert.pem", "hostname":"orderer"}
],
"COMMENT_channels":"fabric-sdk-node Client class requires channel information to be configured during bootstrap.",
"channels": [
{ "name":"mychannel", "peersIndex":[0], "orderersIndex":[0] }
],
"channels-first-network": [
{ "name":"mychannel", "peersIndex":[0,1,2,3], "orderersIndex":[0] }
]
}
}
Once started the Hyperledger Fabric SDK REST server at https://0.0.0.0:3000, when I try to make the GET channels query from the explorer, I get the following error:
error: [fabricconnector.js]: Failed to queryChannels: Error: 14 UNAVAILABLE: Connect Failed
Error not handled for the GET request /api/fabric/1_0/channels: Error: 14 UNAVAILABLE: Connect Failed
at Object.exports.createStatusError ($HOME/mynetwork/fabric-sdk-rest/packages/loopback-connector-fabric/node_modules/grpc/src/common.js:87:15)
at Object.onReceiveStatus ($HOME/mynetwork/fabric-sdk-rest/packages/loopback-connector-fabric/node_modules/grpc/src/client_interceptors.js:1214:28)
at InterceptingListener._callNext ($HOME/mynetwork/fabric-sdk-rest/packages/loopback-connector-fabric/node_modules/grpc/src/client_interceptors.js:590:42)
at InterceptingListener.onReceiveStatus ($HOME/mynetwork/fabric-sdk-rest/packages/loopback-connector-fabric/node_modules/grpc/src/client_interceptors.js:640:8)
at callback ($HOME/mynetwork/fabric-sdk-rest/packages/loopback-connector-fabric/node_modules/grpc/src/client_interceptors.js:867:24)
E0510 10:51:04.780559355 12247 ssl_transport_security.cc:989] Handshake failed with fatal error SSL_ERROR_SSL: error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed.
Has anyone ever seen this error? Can anyone help me get through this, please?

ASP.NET 5: Error with Nuget package in Class library Package

I start playing with VS2015 community version and ASP.NET 5.
I have added a new class library (Package) as a new project into my solution but am unable to install a NuGet package in it. After the restoring, I have a "Reference (Errors - see Error List)".
If i add a basic Library Class (.dll) as a new project into my solution and try to install the same NuGet package, it works beautifully as it used to.
Perhaps, I am misunderstanding how this new class library package works and I am trying to misuse it...
Thanks for your help.
--------------------------------------------------------------EDIT 1-----------------------------------------------------------
Project.json: The warning flag in that case is on the EF package:
{
"version": "1.0.0-*",
"description": "testFinal Class Library",
"authors": [ "SylvainC" ],
"tags": [ "" ],
"projectUrl": "",
"licenseUrl": "",
"dependencies": {
"System.Collections": "4.0.10-beta-23019",
"System.Linq": "4.0.0-beta-23019",
"System.Threading": "4.0.10-beta-23019",
"System.Runtime": "4.0.10-beta-23019",
"Microsoft.CSharp": "4.0.0-beta-23019",
"EntityFramework": "6.1.3"
},
"frameworks": {
"dotnet": { }
}
}
----------------------------------------------------EDIT 1 bis after update to beta8--------------------------
{
"version": "1.0.0-*",
"description": "ClassLibrary5 Class Library",
"authors": [ "SylvainC" ],
"tags": [ "" ],
"projectUrl": "",
"licenseUrl": "",
"frameworks": {
"dnx451": { }
},
"dependencies": {
"EntityFramework": "6.1.3"
}
}
--------------------------------------------------------------EDIT 2-----------------------------------------------------------
After upgrading the dnvm (dvnm upgrade), i was able to run the "dnu restore" command but got the following error:
>dnu restore
Microsoft .NET Development Utility Clr-x86-1.0.0-beta8-15858
GET https://www.nuget.org/api/v2/
GET https://www.myget.org/F/aspnetmaster/api/v2/
Restoring packages for D:\testFinal\project.json
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Reflection'
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='System.Reflection'
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Globalization'
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='System.Globalization'
GET https://www.nuget.org/api/v2/FindPackagesById()?id='EntityFramework'
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='EntityFramework'
Warning: FindPackagesById: System.Globalization
An error occurred while sending the request.
Warning: FindPackagesById: System.Globalization
An error occurred while sending the request.
Warning: FindPackagesById: System.Reflection
An error occurred while sending the request.
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='System.Globalization'
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='System.Reflection'
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Globalization'
Warning: FindPackagesById: System.Reflection
An error occurred while sending the request.
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Reflection'
Warning: FindPackagesById: EntityFramework
An error occurred while sending the request.
Warning: FindPackagesById: EntityFramework
An error occurred while sending the request.
GET https://www.nuget.org/api/v2/FindPackagesById()?id='EntityFramework'
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='EntityFramework'
Warning: FindPackagesById: System.Reflection
An error occurred while sending the request.
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='System.Reflection'
Warning: FindPackagesById: System.Globalization
An error occurred while sending the request.
Warning: FindPackagesById: System.Globalization
An error occurred while sending the request.
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Globalization'
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='System.Globalization'
Warning: FindPackagesById: System.Reflection
An error occurred while sending the request.
GET https://www.nuget.org/api/v2/FindPackagesById()?id='System.Reflection'
Warning: FindPackagesById: EntityFramework
An error occurred while sending the request.
GET https://www.nuget.org/api/v2/FindPackagesById()?id='EntityFramework'
Warning: FindPackagesById: EntityFramework
An error occurred while sending the request.
GET https://www.myget.org/F/aspnetmaster/api/v2/FindPackagesById()?id='EntityFramework'
Error: FindPackagesById: System.Reflection
An error occurred while sending the request.
----------
Error: FindPackagesById: System.Globalization
An error occurred while sending the request.
Error: FindPackagesById: System.Globalization
An error occurred while sending the request.
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: Unable to connect to the remote server ---> System.Net.Sockets.SocketException: No connection could be made because the target machine actively refused it 127.0.0.1:8888
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
---------------------------------------------EDIT 3 with final error and feeds------------------------------
Restore failed
An error occurred while sending the request.
NuGet Config files used:
C:\Users\sylvainc\AppData\Roaming\NuGet\nuget.config
Feeds used:
https://www.nuget.org/api/v2/
https://www.myget.org/F/aspnetmaster/api/v2/
Sylvain
It seems to be a proxy issue giving me this No connection could be made because the target machine actively refused it 127.0.0.1:8888 error.
Tracking it down with Fiddler, I discovered that after opening Fiddler, the restore process and adding new Nuget packages was working fine. A situation quite similar to this post Why calling API in ASP.NET works only if I have Fiddler open? or here: http://www.telerik.com/blogs/help!-running-fiddler-fixes-my-app-.
I also tried to create a dnx.exe.config into C:\Users\sylvainc\.dnx\runtimes\dnx-clr-win-x86.1.0.0-beta8\bin as suggested in http://github.com/aspnet/dnx/issues/451.
Unfortunately it seems not to work with the beta8 version.
My first workaround solution was to use the HTTP protocol to access my NuGet packages as suggested here: Visual Studio 2015 - Can't sign in, Use NuGet, etc. behind corporate proxy.
My final solution was to edit the machine.config file as administrator with Notepad+ in C:\Windows\Microsoft.NET\Framework\v4.0.30319\Config to get ride of the fiddler default proxy definition and to use the system ones:
<!--<system.net>
<defaultProxy enabled="true" useDefaultCredentials="true">
<proxy autoDetect="false" bypassonlocal="false" proxyaddress="http://127.0.0.1:8888" usesystemdefault="false"/>
</defaultProxy>
</system.net>-->
<system.net>
<defaultProxy useDefaultCredentials="true" enabled="true">
<proxy usesystemdefault="True"/>
</defaultProxy>
</system.net>
Now ASP.Net 5 works fine, I even have my intellisense working properly now which was not the case before with the other proxy definition...
Thanks
Sylvain
Be sure you installed the latest tooling for Visual studio. Your project.json seems to be wrong.
Check your nuget feed in visual studio for invalid feed and remove these sources.
Check your active dnx (*)
dnvm list
Active Version Runtime Architecture OperatingSystem Alias
1.0.0-beta4 clr x86 win
1.0.0-beta7 clr x64 win
1.0.0-beta7 clr x86 win b7
1.0.0-beta7 coreclr x64 win
1.0.0-beta7 coreclr x86 win
1.0.0-beta8 clr x64 win
* 1.0.0-beta8 clr x86 win aa, b8
1.0.0-beta8 coreclr x64 win
1.0.0-beta8 coreclr x86 win
1.0.0-rc1-15838 clr x86 win rc1a
1.0.0-rc1-16031 clr x86 win default
1.0.0-rc1-16031 coreclr x86 win
With the tools->nuget package manager-package manager for solution
* Select a package source and test if a package source give any errors
* find out in which nuget feed your package is available
At the end of the dnu output there is a list which config and feeds are used
NuGet Config files used:
C:\Users\Thom\AppData\Roaming\NuGet\nuget.config
Feeds used:
https://api.nuget.org/v3-flatcontainer/
https://www.myget.org/F/aspnetrelease/api/v3/flatcontainer/
Use the dnu restore -? to find the source of the problem.
try this
dnu restore --no-cache --packages .\packages1
Try a package.json like this
{
"version": "1.0.0-*",
"description": "ClassLibrary3 Class Library",
"authors": [ "Thom" ],
"tags": [ "" ],
"projectUrl": "",
"licenseUrl": "",
"dependencies": {
"EntityFramework": "6.1.3"
},
"frameworks": {
"dnx451": { },
}
}
In my case, the package source was pointing to http://packagesource. Changing it to https://www.nuget.org/api/v2/ fixed the probmel