I am working on a repository where I have to extract some features from millions of files.
For me, the current workflow is:
Write code in IntelliJ
Run unit tests
Dry run with small data
ssh to the remote machine
sftp the current code to the remote machine
Run on the server with all million of files
Look into the log exceptions and find out where the code is failing for edge cases
Fix those issues and repeat from step 1
My question would be three-fold:
Is there an easy way how I can sync code with the remote machine automatically (I know I commit to git and then pull the changes in the machine. But is there some other way other than setting up rsync etc.?)
Can I run code directly in the remote machine from IDE and debug it that way?
1) There are a lot of ways to sync your code. Sometimes the best way is to create a kind of deploy script in Python or sh if you don't want to commit and push any changes you are going to test on server. You can use sftp or scp with more automation here (use gzip and so on). Git and rsync are more mature solutions here. But with VCS your problems will be more reproducible and easier to find.
2) You can connect to remote process to debug it directly from IntelliJ. There is a official tutorial for that: https://www.jetbrains.com/help/idea/tutorial-remote-debug.html (but it depends on your security settings because Java debug protocol is not secured itself, you may need to setup SSH tunnel for that)
3) Another, a bit more radical option: you might run IntelliJ IDEA itself on server and debug directly. You can use Projector - an open source project which lets you run IntelliJ on the server with UI in the browser (no X11 required to run). I recommend you look at this repo firstly https://github.com/JetBrains/projector-docker or configure IntelliJ on server following instructions here: https://github.com/JetBrains/projector-server.
Related
Deployments from IDEA are possible, e.g. I can connect to remote server and I can upload a package. The deployment process I have involves a little bit more than just deploying a file on server.
Once the file is uploaded I need to run certain commands to complete the deployment which mean I need to connect to the server and run those commands one by one and I am looking to do that through one click from the IntelliJ IDEA.
Is that possible?
This is how the project directory looks like:
This is how I am trying to setup deployment:
if you click on that little "Fix" button that's what I see
Even if I try to remove the unwanted entries they come back and never go away.
Finally this is what my project structure looks like in idea
Yes, you can use Remote SSH External tools. Make a script that will run your commands, deploy it to the server as well, then run this script remotely via a tool.
In the Run/Debug configuration Before Launch steps you can add your remote external tool to run automatically.
You can find some more details in my another answer.
I have read what the Docker is but having hard time finding of what are the real scenarios of using Docker?
It would be great to see here your usages.
I'm replicating production environment with it, on commit on project with jenkins after building binaries i deploy there, launch the required daemons and run integration tests, all in a very short time (a few seconds over the time that takes the integration tests). Having no need to boot, and little overhead on memory/cpu/disk is great for that kind of things.
I could extend that use for development (just adding a volume where the code resides to my git repository, at least for scripting languages) to have the production environment with the code im actually editing, at a fraction of what virtualbox would require.
Also needed to test how to integrate some 3rd party code into a production system that modified DB. Cloned the DB in a container, installed the production system in another, launched both and iterated the integration until i did it well, going back to zero to try again in seconds, and faster, cheaper and more scriptable than doing it with VMs+snapshots.
Also run several desktop browser instances on containers, with their own plugins, cookies, data storage and so on separated. The docker repository example for desktop integration is a good start for it, but planning to test subuser to extend this kind of usage.
I've used Docker to implement a virtualized build server which any user could ask to run a build off their personal git branch in our canonical environment.
Each SSH connection made to the server was connected to a new container, ensuring that all builds were isolated from each other (a major pain point in the past), ensuring that the container's state couldn't be corrupted (since changes were all isolated to that single instance), and ensuring that even developers on platforms such as Windows where Docker (and other tools in our canonical build environment) couldn't be run locally would be able to run builds.
We use it for the following uses:
We have a Jenkins Container which we can use to bring up our Jenkins server. We mount the workspace using volumes so we can migrate the server easily just by copying the files and launching the container somewhere else.
We use a Jetty container to easily deploy our war files in our production and development environment.
We use a whole host of other monitoring tools such as Uptime which we have containers for so that we can bring them up and down on various hosts with a single command.
I use docker to build and test our software on several different Linux distributions (RHEL 4/5/6/7, Ubuntu 12.04, 14.04).
Docker makes it easy and fast to create minimalistic and consistent build environments.
Docker gives you the benefits that other virtualization solutions give you to a fraction of the recourse needed.
I'm trying to deploy openMRS v.1.9.2 to a local VM running CentOS & Glassfish 2 for work. Unfortunately, I could not get it to work. Normally, I just download the standalone found at source forge. I just double-click the jar, and I'm good to go.
I normally just SSH into the the VM, so I first tried doing everything through a terminal. Here are the steps I took:
Using wget, retrieve the .zip
Create a dir (I just called it /openmrs), cd into the new directory, and then expand the .zip.
cd into the directory.
At this point, there are two options to start openMRS.
Run the bash script: ./run-on-linux.sh
Run the .JAR: java -jar [insert_jar_name].jar -commandline
When I run the .JAR, I get a stack trace.
When I try to run the bash script, I get another error.
Anyways, I thought I found a potential solution in an openMRS JIRA ticket, but it seems aimed at Glassfish 3, and not Glassfish 2 (which is what I need to use).
I then tried deploying the .WAR via the Glassfish admin UI. I thought it would work, but after going through the steps of selecting a language, whether or not to use demo data, etc. I received this.
Does anyone have experience deploying openMRS to Glassfish 2.1.1? Unfortunately Glassfish 3 doesn't seem to be a realistic option. I would really appreciate any help here. Thanks.
Although it doesn't solve my problem of not being able to successfully deploy openMRS to an instance of Glassfish v.2, I did manage to get myself further by just installing MySQL on the VM. Our work machines are all set up for postgres, so I think should have guessed earlier that not having a MySQL server installation was the problem.
Here is a tutorial I used to install MySQL
I am looking for something similar to RStudio-server (r-studio), but for programming in Java/Scala.
RStudio-server accessed via web is the same as the desktop version.
May be there are other equivalent solutions, given that my problem is scarce local resources to run my programs, but plenty of resources in a remote computer.
The more seamless, the better.
It would be also good if I could just run my program remotely seamlessly (outputing like if it was local and inside the IDE).
I recently starting using Chromoting to remote into a different machine that has IntelliJ. It's convenient in that it's built into chrome and I can pretty much do this anywhere because everyone has Chrome these days. It's a very good remoting tool but it still is remoting, it has its delays and minor problems (some keyboard shortcuts don't get sent through and those options haven't been added yet).
Just an idea for something to try.
It is not seamless, but I just managed to solve the problem using the right workflow:
In the local machine:
edit files in IDEA
[optional] set the output path, if any, to your favorite file hosting service (dropbox in my case)
commit with your favorite VCS (bzr in my case, it can need a push to the remote machine)
In the remote machine:
update with your favorite VCS
compile and run with your favorite build tool (sbt in my case):
sbt 'run-main experiments.AtomicBombMain'
[optional] In the local machine again:
[optional] get the generated files at the output path at the shared folder of your favorite file hosting service
Link with more details about the entire process since instalation until compiling:
sbt-intellij-idea-scala-debian-wheezy-how-to
Suppose I have a server that runs on Linux on which I would like to develop software (mainly OCaml, C/C++ and Java).
Is there a way to "remote develop" these things? I mean an IDE that allows me to modify files remotely (they are then uploaded when modified and saved) and to compile through SSH (basically invoking make or omake).
I was looking for something that makes this process transparent to the developer, without caring of doing things by hand. I'm used to use Eclipse so I wonder if a plugin to achieve this exists or if are there other choices?
Mind that it may happen that the local machine it not able to build software I intend to (for example for OCaml) so it should rely just on remote connection.
Thanks in advance
You can use X11 forwarding. Even if you are connecting from a Windows machine.
If you are on Linux, connecting with ssh -Y might work right out of the box for you:
ssh -Y user#your_server
eclipse &
Well the simplest idea I can think of, though it is rather brute force would be to just open up a file share to the server and then edit the file directly through Eclipse.
If that doesn't work for Java at least you could make use of Maven to do some of those tasks. I am less certain about invoking Make though.
I think your answer is IDE-centric.
KDE's ioslaves support access over both SFTP and SSH (using fish, which uses a Perl script uploaded to the remote machine). I believe Gnome also has a virtual file system (gvfs) which supports remote filesystem access.
My recommendation, therefore, is to choose an IDE which supports a virtual filesystem that can operate over SSH/SFTP and allows you to specify the build command. You would then only need to specify the build command which would get its output from the remote make command (for example, vim has a makeprg option which can be set to any arbitrary command).
Depending on how 'remote' this is; why not ssh in and run the IDE remotely over X?
Using a build tool (Hudson for example) you could put a build agent on your remote server, check your changes into your repository as normal, and have it do a build when you check in changes (it will either do a repository hook or poll for changes, probably). Your build process will be the same, it will simply be automated. :-)
emacs has tramp, which lets you both open and save remote files, and open a shell on a remote system. Working with tramp is almost exactly like working with local files, except for the filename. To open 'foo.c' on the machine 'bork' as user 'joe' I open it with the standard emacs commands, giving it the pathname /joe#bork:foo.c
I use vim for remote development. (Well, I use vim also non-remote.)
If building is the problem, have you thought about simply using an automated build system where you commit to svn and the system then automatically builds the software? I've heard many good things about these sorts of systems, although I haven't quite tried any out myself.
As for remote development, a SVnDAV solution might be reasonable. It basically commits your every save and is completely transparent to the text editor you're using. However a probably much nicer solution would simply be to use a networked drive/directory and edit files remotely. On all unix-based systems this should work completely transparently to both the developer and the text editor.
Your choice of IDE will have the most impact on the answer to "can I?". If your IDE of choice is CLI based than you can always just SSH in, fire up screen (so that your CLI session is persistent across SSH sessions), and have at it!
Use vim or emacs since they will offer you speed. I know there is a learning curve associated with these editors; but once you get comfortable in any of them; you will be able to work on them as good as with Eclipse or any other IDE.
If you already have a linux server then I would suggest setting up a simple VPN server. I have done this in the past and it works pretty well. This way you can connect and modify/build your files with any "local" OS. I did this cause I use mac, pc and linux through various parts of the day and in multiple locations, so the VPN allowed me to edit files remotely w/out having to allow file sharing over the internet.
There are plenty of tutorials about how to achieve this even if you are newer to linux. I use ubuntu server on my linux box and here are the tutorial I have used.
http://www.ubuntugeek.com/howto-pptp-vpn-server-with-ubuntu-10-04-lucid-lynx.html
Netbeans 7.3 has a new feature which addresses your problem (and mine). Here's the tutorial.
https://netbeans.org/kb/docs/cnd/remotedev-tutorial.html
note: I realize it has been 3 years since this question was asked so the answer may be irrelevant to #Jack now.
One IDE that supports exactly your language set is Nuclide. It adds some packages to Atom and is used internally in Facebook exactly as you have described - full-fledged remote development in C++, Java, and Ocaml.
If a friendly file editor is enough for you then I'd recommend to use Jupyter.
Super fast installation
Built in server/file editor that starts with one command