my team is building a flink based realtime computation platform. We submit flink job to Yarn.
We create a Process and run commit command use CLI. In order to get yarn application id, we create a thread and parse process output. Application id is used in other methods.
For example, we submit job by this command:
nohup flink run -m yarn-cluster -d -yqu root.default
-ynm BDP_RTC_FLINK_10457_MultiOutputTestFrontEnd -yjm 1024
-yn 2 -ytm 1024 -ys 2
The output is shown below:
2018-10-10 11:21:04 [info] 2018-10-10 11:21:04,629 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Submitting application master application_1536669298614_67675
2018-10-10 11:21:04 [info] 2018-10-10 11:21:04,654 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl - Submitted application application_1536669298614_67675
2018-10-10 11:21:04 [info] 2018-10-10 11:21:04,656 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - Deploying cluster, current state ACCEPTED
2018-10-10 11:21:12 [info] 2018-10-10 11:21:12,699 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - YARN application has been deployed successfully.
2018-10-10 11:21:12 [info] 2018-10-10 11:21:12,700 INFO org.apache.flink.yarn.AbstractYarnClusterDescriptor - The Flink YARN client has been started in detached mode.
We parse process output and get application id: application_1536669298614_67675.
Are there any other elegant solutions to get application id in our situation?
Maybe you can get the relation between the yarn application and the flink job.
Firstly, list the yarn application.
yarn application -list
Then, you get the application list, and you can list flink job on the yarn application.
./bin/flink list -m yarn-cluster -yid <Yarn Application Id>
By the way, you can use
./bin/flink run -d
not use
nohup
Related
We are facing issues with React-Native in AppCenter. We are Once I run the build, everything goes well but at end we are facing below issue.
No addition pre or post script added in AppCenter.
Execution optimizations have been disabled for 12 invalid unit(s) of work during this build to ensure correctness.
Please consult deprecation warnings for more details.
**BUILD SUCCESSFUL** in 11m 32s
362 actionable tasks: 351 executed, 11 up-to-date
##[section]Finishing: Gradle Task
##[section]Starting: Android Postprocess
==============================================================================
Task : Shell script
Description : Run a shell script using Bash
Version : 2.212.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/shell-script
==============================================================================
[command]/bin/bash /Users/runner/runners/2.214.1/scripts/android-postprocess.sh /Users/runner/work/1/s/android/app/build/outputs/apk
Removing all ABI or density dependent APKs…
Found 0 APK file(s)
Found 0 unaligned APK file(s)
~/work/1/s/android/app/build/outputs ~/runners/2.214.1/scripts
~/runners/2.214.1/scripts
##[section]Async Command Start: Add Build Tag
Build '16' has following tags now: normal, android-app-bundle, javascript, android, manual, signed, mapping, distribution
##[section]Async Command End: Add Build Tag
##[section]Finishing: Android Postprocess
##[section]Starting: Sign APK
==============================================================================
Task : Android signing
Description : Sign and align Android APK files
Version : 3.201.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/android-signing
==============================================================================
##****[error]Error: No matching files were found with search pattern: ****/Users/runner/work/1/s/android/app/build/**/*.apk
##[section]Finishing: Sign APK
##[section]Starting: Sign APK
==============================================================================
Task : Android signing
Description : Sign and align Android APK files
Version : 3.201.0
Author : Microsoft Corporation
Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/android-signing
==============================================================================
##[section]Finishing: Sign APK
##[section]Starting: Checkout XXXXXXX#development to s
==============================================================================
Task : Get sources
Description : Get sources from a repository. Supports Git, TfsVC, and SVN repositories.
Version : 1.0.0
Author : Microsoft
Help : [More Information](https://go.microsoft.com/fwlink/?LinkId=798199)
==============================================================================
Cleaning any cached credential from repository: XXXXXXX (ExternalGit)
##[section]Finishing: Checkout XXXXXXX#development to s
##[section]Starting: Finalize Job
Cleaning up task key
Start cleaning up orphan processes.
Terminate orphan process: pid (23181) (java)
Terminate orphan process: pid (22867) (java)
Terminate orphan process: pid (11113) (java)
Terminate orphan process: pid (7987) (java)
##[section]Finishing: Finalize Job
##[section]Finishing: Build
I tried with all the options but unable to resolve the issues.
I got the answer of my own question. As per Appcenter document you need to set universalApk to true. This is the only way in Appcenter weather you plan to make multiple apk to reduce the size.
Hello~ I am trying to use Github workflow to run the unit test for the code in my repository.
So I wrote a yaml file, its function is when I push my code to my repository, it can let me use my local environment to execute my code on Github, and the purpose of these codes is to run the unit tests.
But I can't run this workflow successfully, this error message always appears. I'm curious why I can execute unit test successfully on the local IDE, but it doesn't work when I use workflow to automatically execute it for me.
============================== warnings summary ===============================
..\..\..\..\anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8
C:\Users\COA\anaconda3\lib\site-packages\pyreadline\py3k_compat.py:8: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
return isinstance(x, collections.Callable)
-- Docs: https://docs.pytest.org/en/stable/warnings.html
=========================== short test summary info ===========================
ERROR test/test_createds_for_fnn.py - AttributeError: type object 'h5py.h5.H5...
ERROR test/test_fnn.py - AttributeError: type object 'h5py.h5.H5PYConfig' has...
ERROR test/test_unet.py - AttributeError: type object 'h5py.h5.H5PYConfig' ha...
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!
======================== 1 warning, 3 errors in 2.76s =========================
Error: Process completed with exit code 1.
My environment:
Windows 10 OS
Python 3.8.8
Tensorflow-gpu 2.7.0
pytest 6.2.3
h5py 3.6.0
Workflow
# This is a basic workflow to help you get started with Actions
name: CI
# Controls when the workflow will run
on:
# Triggers the workflow on push or pull request events but only for the master branch
push:
branches: [ main ]
pull_request:
branches: [ main ]
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
# A workflow run is made up of one or more jobs that can run sequentially or in parallel
jobs:
# This workflow contains a single job called "build"
build:
# The type of runner that the job will run on
runs-on: self-hosted
# Steps represent a sequence of tasks that will be executed as part of the job
steps:
# Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
- uses: actions/checkout#v3
# Runs a single command using the runners shell
- name: Run a one-line script
run: echo Hello, world!
# Runs a set of commands using the runners shell
- name: Run a multi-line script
run: |
echo Add other actions to build,
echo test, and deploy your project
echo train the fnn
pytest
Using filebeat 7.5.2:
I'm using a filebeat configuration with close_eof enabled and I run filebeat with the flag --once. I can see the harvester reaching eof but the filebeat keeps going.
Flebeat conf:
filebeat.inputs:
- type: log
close_eof: true
enabled: true
paths:
- "${LOGS_PATH}"
scan_frequency: 1s
fields: {
machine: "${HOST}"
}
output.logstash:
hosts: ["192.168.41.6:5044"]
bulk_max_size: 1024
timeout: 30s
pipelining: 1
workers: 1
And I run it using:
filebeat run --once -v -c "PATH TO CONF..."
And some logs from the filebeat instance:
...
2020-02-04T18:30:16.950Z INFO instance/beat.go:297 Setup Beat: filebeat; Version: 7.5.2
2020-02-04T18:30:17.059Z INFO [publisher] pipeline/module.go:97 Beat name: logstash
2020-02-04T18:30:17.167Z WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch out
put is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-04T18:30:17.168Z INFO instance/beat.go:429 filebeat start running.
2020-02-04T18:30:17.168Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-04T18:30:17.168Z INFO registrar/migrate.go:104 No registry home found. Create: /tmp/tmp.BXJtfiaEzb/data/registry/filebeat
2020-02-04T18:30:17.179Z INFO registrar/migrate.go:112 Initialize registry meta file
2020-02-04T18:30:17.192Z INFO registrar/registrar.go:108 No registry file found under: /tmp/tmp.BXJtfiaEzb/data/registry/filebeat/data.json. Creating a new re
gistry file.
2020-02-04T18:30:17.193Z INFO registrar/registrar.go:145 Loading registrar data from /tmp/tmp.BXJtfiaEzb/data/registry/filebeat/data.json
2020-02-04T18:30:17.193Z INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-02-04T18:30:17.193Z WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch out
put is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-04T18:30:17.193Z INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-04T18:30:17.194Z INFO log/input.go:152 Configured paths: [/tmp/tmp.BXJtfiaEzb/*.log]
2020-02-04T18:30:17.206Z INFO input/input.go:114 Starting input of type: log; ID: 13918413832820009056
2020-02-04T18:30:17.225Z INFO input/input.go:167 Stopping Input: 13918413832820009056
2020-02-04T18:30:17.225Z INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-02-04T18:30:17.225Z INFO log/harvester.go:251 Harvester started for file: /tmp/tmp.BXJtfiaEzb/dcbgw-20200124080032_darkblue.log
2020-02-04T18:30:17.231Z INFO beater/filebeat.go:384 Running filebeat once. Waiting for completion ...
2020-02-04T18:30:17.231Z INFO beater/filebeat.go:386 All data collection completed. Shutting down.
2020-02-04T18:30:17.231Z INFO crawler/crawler.go:139 Stopping Crawler
2020-02-04T18:30:17.231Z INFO crawler/crawler.go:149 Stopping 1 inputs
2020-02-04T18:30:17.258Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://192.168.41.6:5044))
2020-02-04T18:30:17.296Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://192.168.41.6:5044)) established
... Only metrics here ...
2020-02-04T18:35:55.686Z INFO log/harvester.go:274 End of file reached: /tmp/tmp.BXJtfiaEzb/dcbgw-20200124080032_darkblue.log. Closing because close_eof is enabled.
2020-02-04T18:35:55.686Z INFO crawler/crawler.go:165 Crawler stopped
... MORE METRICS ...
2020-02-04T18:36:26.609Z ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 192.168.41.6:49662->192.168.41.6:5044: i/o timeout
2020-02-04T18:36:26.621Z ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected
2020-02-04T18:36:28.520Z ERROR pipeline/output.go:121 Failed to publish events: client is not connected
2020-02-04T18:36:28.520Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://192.168.41.6:5044))
2020-02-04T18:36:28.521Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://192.168.41.6:5044)) established
... MORE METRICS ...
From this I'm outputing this to Logstash 7.5.2 running in the same Ubuntu 18 VM. Running Logstash with log level trace does not output any error.
Hal Version
root#9d4ea3d3c90e:/workdir# hal --version
0.43.0-180317140630
Applying config with hal
root#9d4ea3d3c90e:/workdir# hal deploy apply
+ Get current deployment
Success
+ Prep deployment
Success
Problems in default.provider.kubernetes.sandbox-spinnaker:
- WARNING Field KubernetesAccount.kinds not supported for Spinnaker
version 1.6.0: Configuring kind caching behavior is not supported yet.
? Use at least 1.7.0 (It may not have been released yet).
- WARNING Field KubernetesAccount.omitKinds not supported for
Spinnaker version 1.6.0: Configuring kind caching behavior is not supported
yet.
? Use at least 1.7.0 (It may not have been released yet).
+ Preparation complete... deploying Spinnaker
+ Get current deployment
Success
+ Apply deployment
Success
+ Run `hal deploy connect` to connect to Spinnaker.
Status of pods
$ kubectl get pods
NAME READY STATUS RESTARTS AGE
spin-clouddriver-f54b887f-hmnsx 1/1 Running 0 19h
spin-deck-598f8df868-6jlhl 1/1 Running 0 19h
spin-echo-84d9b7d679-whv6n 1/1 Running 0 19h
spin-fiat-78b899d45d-dk767 0/1 CrashLoopBackOff 4 3m
spin-front50-65d476986f-hgnsq 1/1 Running 0 19h
spin-gate-67b6955476-b7dhj 1/1 Running 0 19h
spin-igor-56f7f9b85f-rj7fl 1/1 Running 0 19h
spin-orca-6c4f48cf49-5knlc 1/1 Running 0 19h
spin-redis-69b6cc4c46-vmfhr 1/1 Running 0 5d
Log message from spin-fiat
$ kubectl logs spin-fiat-78b899d45d-dk767
...
2018-03-21 17:16:10.472 ERROR 1 --- [ main] o.s.boot.SpringApplication : [] Application startup failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'resourcesConfig': Injection of autowired dependencies failed; nested exception is java.lang.IllegalArgumentException: Could not resolve placeholder 'services.front50.baseUrl' in value "${services.front50.baseUrl}"
I'm not sure why spin-fiat is not starting.
Any tips on how to troubleshoot appreciated.
This issue is fixed in PR#1511
I suggest you upgrade your Spinnaker to version 1.20.X. That version uses the new Kubernetes V2 provider. Also the FIAT issue is gone. You must install or upgrade halyard for this new Spinnaker Release.
Upgrade Halyard
Upgrade Spinnaker to version 1.20.X
I have created a simple Spring MVC app and built and deployed using apache tomcat 7 on my localhost. The build for the same app is failing on Travis CI and the build log reads :
$ git clone --depth=50 --branch=master git://github.com/swarup-donepudi/BidToBuy.git swarup-donepudi/BidToBuy
Cloning into 'swarup-donepudi/BidToBuy'...
remote: Counting objects: 32, done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 32 (delta 7), reused 28 (delta 3), pack-reused 0
Receiving objects: 100% (32/32), 5.35 KiB | 0 bytes/s, done.
Resolving deltas: 100% (7/7), done.
Checking connectivity... done.
$ cd swarup-donepudi/BidToBuy
$ git checkout -qf 4faeebe46b78137ea5676f49e2bf5224a8ffcb90
This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.
If you require sudo, add 'sudo: required' to your .travis.yml
See http://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.
$ java -version
java version "1.7.0_76"
Java(TM) SE Runtime Environment (build 1.7.0_76-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.76-b04, mixed mode)
$ javac -version
javac 1.7.0_76
0.92s$ ant deps
Buildfile: /home/travis/build/swarup-donepudi/BidToBuy/build.xml
BUILD FAILED
/home/travis/build/swarup-donepudi/BidToBuy/build.xml:87: /home/travis/build/swarup-donepudi/BidToBuy/C:/tomcat/lib does not exist.
Total time: 0 seconds
The command "ant deps" failed and exited with 1 during .
Your build has been stopped.
The build.properties file :
# Ant properties for building the springapp
appserver.home=C:/tomcat
# for Tomcat 5 use $appserver.home}/server/lib
# for Tomcat 6 use $appserver.home}/lib
appserver.lib=C:/tomcat/lib
deploy.path=C:/tomcat/webapps
tomcat.manager.url=http://localhost:8091/manager/text
tomcat.manager.username=tomcat
tomcat.manager.password=s3cret
Contents of .travis.yml file:
language: java
sudo: false
install: ant deps
I am assuming that the possible reason could be the path of the tomcat lib folder.
How does travis ci handle builds configured for tomcat server? I tried to find the answer but only to find none. Please help
I guess that the path you have given is not valid for the Travis CI environment:
/home/travis/build/swarup-donepudi/BidToBuy/C:/tomcat/lib does not exist
As you are stating this in your properties file:
C:/tomcat
Take into account that Travis CI runs the build within a Linux Container, so you will have to explore other ways to deploy your application.
As you are using Spring, perhaps you should try Spring Boot, as it comes with an embedded Tomcat server, therefore freeing you from the build script.