Not able to download .csv file through Google Chrome in headless mode - selenium

Question

I have been trying to download a csv file using selenium in Chrome headless mode but it is not working for my scenario. I tried to run the same script without headless and it works. I tried the same headless configuration with https://www.mockaroo.com/ site and it worked. Below is the chrome headless configuration that I'm using
System.setProperty("webdriver.chrome.driver", Constants.CHROME_DRIVER);
ChromeOptions options = new ChromeOptions();
options.addArguments("--test-type");
options.addArguments("--headless");
options.addArguments("--disable-extensions"); //to disable browser extension popup
options.addArguments("window-size=1980,1080");
options.addArguments("--disable-web-security");
options.addArguments("--no-sandbox");
options.addArguments("--disable-gpu");
options.addArguments("--disable-popup-blocking");
options.addArguments("--start-maximized");
options.addArguments("--allow-running-insecure-content");
options.setExperimentalOption("w3c", false);
options.addArguments("--ignore-certificate-errors");
options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36");
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(ChromeOptions.CAPABILITY, options);
// set performance logger
// this sends Network.enable to chromedriver
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
cap.setCapability("goog:loggingPrefs", logPrefs);
ChromeDriverService driverService = ChromeDriverService.createDefaultService();
driver = new ChromeDriver(driverService, cap);
String downloadFilepath = "C:\\Users\\User\\Downloads";
System.out.println(downloadFilepath);
Map < String, Object > commandParams = new HashMap<String, Object>();
commandParams.put("cmd", "Page.setDownloadBehavior");
Map < String, Object > params = new HashMap<String, Object>();
params.put("behavior", "allow");
params.put("download.prompt_for_download", false);
params.put("downloadPath", downloadFilepath);
commandParams.put("params", params);
ObjectMapper objectMapper = new ObjectMapper();
HttpClient httpClient = HttpClientBuilder.create().build();
try {
String command = objectMapper.writeValueAsString(commandParams);
String u = driverService.getUrl().toString() + "/session/" + ((ChromeDriver)driver).getSessionId() + "/chromium/send_command";
System.out.println("U: " + u.toString());
HttpPost request = new HttpPost(u);
request.addHeader("content-type", "application/json");
request.setEntity(new StringEntity(command));
httpClient.execute(request);
} catch (Exception e) { }
driver.get(prop.getProperty("url"));
With the above driver configuration (after authenticating the user), I'm clicking a button which submits the form data and in response, I am getting:
Content-Disposition: attachment;filename=WIS_mop_mis3.csv
which downloads the file.
Please find the logs below where you can see the highlighted part where it says that net::ERR_ABORTED but I am getting this error for both headless & without headless;
Performance logs for headless mode
Anyone have any idea, where I am doing wrong.
Edit: The application flow is when user clicks on the export button a new window gets opened where user fills the data and clicks on a button which calls a function for some sort of validation. After validation, it submits the form using document.formName.submit().

Answer 1

I am able to figure out the solution. Below is the working code
System.setProperty("webdriver.chrome.driver", Constants.CHROME_DRIVER);
ChromeOptions options = new ChromeOptions();
options.addArguments("--test-type");
options.addArguments("--headless");
options.addArguments("--disable-extensions"); //to disable browser extension popup
options.addArguments("window-size=1980,1080");
options.addArguments("--disable-web-security");
options.addArguments("--no-sandbox");
options.addArguments("--disable-gpu");
options.addArguments("--disable-popup-blocking");
options.addArguments("--start-maximized");
options.addArguments("--allow-running-insecure-content");
options.setExperimentalOption("w3c", false);
options.addArguments("--ignore-certificate-errors");
options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36");
HashMap<String, Object> chromePreferences = new HashMap<String, Object>();
// This part was missing in my old code.
chromePreferences.put("download.prompt_for_download", false);
chromePreferences.put("download.directory_upgrade", true);
chromePreferences.put("safebrowsing.enabled", false);
chromePreferences.put("profile.default_content_settings.popups", 0);
chromePreferences.put("safebrowsing.disable_download_protection", true);
chromePreferences.put("download.default_directory", downloadFilepath);
options.setExperimentalOption("prefs", chromePreferences);
DesiredCapabilities cap = DesiredCapabilities.chrome();
cap.setCapability(ChromeOptions.CAPABILITY, options);
// set performance logger
// this sends Network.enable to chromedriver
LoggingPreferences logPrefs = new LoggingPreferences();
logPrefs.enable(LogType.PERFORMANCE, Level.ALL);
cap.setCapability("goog:loggingPrefs", logPrefs);
ChromeDriverService driverService = ChromeDriverService.createDefaultService();
driver = new ChromeDriver(driverService, cap);
String downloadFilepath = "C:\\Users\\User\\Downloads";
System.out.println(downloadFilepath);
Map < String, Object > commandParams = new HashMap<String, Object>();
commandParams.put("cmd", "Page.setDownloadBehavior");
Map < String, Object > params = new HashMap<String, Object>();
params.put("behavior", "allow");
params.put("downloadPath", downloadFilepath);
commandParams.put("params", params);
ObjectMapper objectMapper = new ObjectMapper();
HttpClient httpClient = HttpClientBuilder.create().build();
try {
String command = objectMapper.writeValueAsString(commandParams);
String u = driverService.getUrl().toString() + "/session/" + ((ChromeDriver)driver).getSessionId() + "/chromium/send_command";
System.out.println("U: " + u.toString());
HttpPost request = new HttpPost(u);
request.addHeader("content-type", "application/json");
request.setEntity(new StringEntity(command));
httpClient.execute(request);
} catch (Exception e) { }
This post helped me to fix this issue https://stackoverflow.com/a/52384942/6017332