Rename & Copy Multiple Files (Different names) to One Name With Numbers - batch-rename

how can I rename or copy multiple files (with different name without numbers like "jhf.mp4" "JJHGF.flv" ..) to one name with numbers like (input_01.mp4 input_02 ....) (order by case) by batch file by giving just path of (folder files) to batch file or dropping the files to it (patch file).
-- MEAN USING VARIABLE OR (dropping with %*) .
Note 1: I need this technique for encoding animes with X264 if any other good ideas I'll be happy to hear it.
Note 2: I'm using Windows 10.

I think you can solve this problem using WSHost-scripts. WSHost-system should be preinstalled on any modern Windows computer.
I designed a system of utilities to tackle your problem:
build-mapping-to-normalized-filenames.js
map-filenames.js
cleanup.cmd
before.cmd
after.cmd
You can find source code for 3-4-5 at the bottom of this answer, and for 1-2 - in separate answers (there is a limit of characters per answer).
1 and 2 are workhorses of this system. Powerful but not user-friendly. Yes you can use them as stand-alone utilities if you understand what input they use.
3-4-5 - are glue that bind this system and expose user-friendly controls.
This system is dependent on absolute paths. But all absolute paths are set as variables at the start of 3-4-5 - so if you want to reorganize system - you need change things there and only there.
This is how it should be deployed at filesystem:
folders:
C:\1\ <- this is the root of system
C:\1\aOrig <- folder for original files
C:\1\norm\ <- folder with files that are part of this system + temp folders
C:\1\norm\backend <- place for non-user-friendly utilities
files:
C:\1\norm\backend\build-mapping-to-normalized-filenames.js
C:\1\norm\backend\map-filenames.js
C:\1\norm\after.cmd
C:\1\norm\before.cmd
C:\1\norm\cleanup.cmd
+files: (these are example international filesnames that you can use for playground to check how system works to make yourserf familiar with it)
C:\1\aOrig\English.txt <- test files, you could add unique content to each
C:\1\aOrig\Русский.txt
C:\1\aOrig\العربية.txt
C:\1\aOrig\ܐܪܡܝܐ.txt
C:\1\aOrig\中文.txt
C:\1\aOrig\日本語.txt
How to use it:
You open your console and go to dir **C:\1\norm**
You call programs in this order:
cleanup.cmd
before.cmd
What happened now is that two interesting temp folders are created:
C:\1\norm\bTemp <- here are normalized files.
C:\1\norm\mappingTemp <- it contains a file with mapping between original and normalized names.
What you must do now - is to process files in C:\1\norm\bTemp - that special processing you mentioned before. Right now for the purposes of learning, you can manually change extensions and contents.
after.cmd
cleanup.cmd
And that's it.
So, if we condence it that's your typical session here, plain and simple -
cleanup.cmd
before.cmd
..do your stuff with files at "C:\1\norm\bTemp"...
after.cmd
cleanup.cmd
Also, you don't need command line to use those - you can open **C:\1\norm** in Explorer and double click those utilities in the order that I explained above - cleanup-before-(..manually do your work with bTemp..)-after-cleanup
Listing for program cleanup.cmd :
#ECHO OFF
:: name of this script...
:: useful for debugging...
SET me=%~nx0
:: aOrig - directory that contains source files...
:: bTemp - temporary working directory...
:: mTemp - temporary directory for mapping.txt...
SET aOrig=C:\1\aOrig
SET bTemp=C:\1\norm\bTemp
SET mTemp=C:\1\norm\mappingTemp
:: Removing temporary folders...
:: Be warned - this RD is working
:: in quiet mode /Q - so it wouldn't ask you politely
:: about del y/n - thus, please make shure that
:: %aOrig% doesn't point to anything
:: important for your personal data or system...
IF EXIST "%bTemp%" RD /S /Q "%bTemp%"
IF EXIST "%mTemp%" RD /S /Q "%mTemp%"
IF %ERRORLEVEL% NEQ 0 (
ECHO Script %me% failed on step cleanup-1: [Removing temporary folders]...
ECHO Error: %ERRORLEVEL%
EXIT /B 1
)
:: Successfully finished
ECHO.
ECHO Script %me% successfully finished!
Listing for program before.cmd :
#ECHO OFF
:: name of this script...
:: useful for debugging...
SET me=%~nx0
:: aOrig - directory that contains original files...
:: bTemp - temporary working directory...
:: mTemp - temporary directory for mapping.txt...
SET aOrig=C:\1\aOrig
SET bTemp=C:\1\norm\bTemp
SET mTemp=C:\1\norm\mappingTemp
:: scriptBuild - name of .js-script for building a map
:: from original to normalized filenames...
:: scriptMapFi - name of .js-script interpreting
:: mapping file that was build by scriptBuild...
:: mappingFile - path for file that will contain mapping between
:: original filenames and normalized filenames...
SET scriptBuild=C:\1\norm\backend\build-mapping-to-normalized-filenames.js
SET scriptMapFi=C:\1\norm\backend\map-filenames.js
SET mappingFile=C:\1\norm\mappingTemp\mapping.txt
:: creating temporary folders...
IF NOT EXIST "%bTemp%" MKDIR "%bTemp%"
IF NOT EXIST "%mTemp%" MKDIR "%mTemp%"
IF %ERRORLEVEL% NEQ 0 (
ECHO Script %me% failed on step before-1: [Creating temporary folders]...
ECHO Error: %ERRORLEVEL%
EXIT /B 1
)
:: now we run build-mapping-to-normalized-filenames.js ...
:: that //U thing is important...
CSCRIPT //U //NoLogo "%scriptBuild%" "%aOrig%" 0 1 > "%mappingFile%"
IF %ERRORLEVEL% NEQ 0 (
ECHO Script %me% failed on step before-2: [Building a list of normalized filenames]...
ECHO Error: %ERRORLEVEL%
EXIT /B 1
)
:: now we run map-filenames.js ...
:: that //U thing is important...
CSCRIPT //U //NoLogo "%scriptMapFi%" "%aOrig%" "%bTemp%" /NotTraining < "%mappingFile%"
IF %ERRORLEVEL% NEQ 0 (
ECHO Script %me% failed on step before-3: [Copy files while simultaneously changing their names to normalised form]...
ECHO Error: %ERRORLEVEL%
EXIT /B 1
)
:: Successfully finished
ECHO.
ECHO Script %me% successfully finished!
Listing for program after.cmd :
#ECHO OFF
:: name of this script...
:: useful for debugging...
SET me=%~nx0
:: aOrig - directory that contains original files...
:: bTemp - temporary working directory...
:: mTemp - temporary directory for mapping.txt...
SET aOrig=C:\1\aOrig
SET bTemp=C:\1\norm\bTemp
SET mTemp=C:\1\norm\mappingTemp
:: scriptBuild - name of .js-script for building a map
:: from original to normalized filenames...
:: scriptMapFi - name of .js-script interpreting
:: mapping file that was build by scriptBuild...
:: mappingFile - path for file that will contain mapping between
:: original filenames and normalized filenames...
SET scriptBuild=C:\1\norm\backend\build-mapping-to-normalized-filenames.js
SET scriptMapFi=C:\1\norm\backend\map-filenames.js
SET mappingFile=C:\1\norm\mappingTemp\mapping.txt
:: Removing files in original folder...
:: Not recursive
:: Be warned - this DEL is working
:: in quiet mode /Q - so it wouldn't ask you politely
:: about del y/n - thus, please make shure that
:: %aOrig% doesn't point to anything
:: important for your personal data or system...
IF EXIST "%aOrig%" DEL /Q "%aOrig%"
IF %ERRORLEVEL% NEQ 0 (
ECHO Script %me% failed on step after-1: [Removing files from original folder]...
ECHO Error: %ERRORLEVEL%
EXIT /B 1
)
:: now we run map-filenames.js ...
:: that //U thing is important...
CSCRIPT //U //NoLogo "%scriptMapFi%" "%bTemp%" "%aOrig%" /NotTraining /MapFrom2To1 /ResultExtension:2 /FuzzyExtensionsFrom < "%mappingFile%"
IF %ERRORLEVEL% NEQ 0 (
ECHO Script %me% failed on step after-2: [Copy files while simultaneously changing their names to normalised form]...
ECHO Error: %ERRORLEVEL%
EXIT /B 1
)
:: Successfully finished
ECHO.
ECHO Script %me% successfully finished!

Listing for program build-mapping-to-normalized-filenames.js :
// - - - - -
// #magicVars #config
// Purpose of basicOptions: organize set of most basic settings
// into a compact object package
var basicOptions = {
// Fist things first -
// "scriptName" will be important for error messages.
// Because when you debug a pipeline of scripts, it is always
// crucial to know - which program in pipeline of 10 utilities
// throws that cryptic stderr message "error: not enough arguments".
// I mean huh? Was that calc-tool.js? Or sort-tool.js? Go figure...
scriptName: WScript.ScriptName.toString(),
// What kind of End Of Line sequence
// (they call it New Line sometimes too)
// you prefer when using outputToolbox?
eolStdOut: '\r\n',
eolStdErr: '\r\n',
// Those are all possible errorCodes that this script
// was designed to fail with.
// You can define other error codes.
// Those SHOULD be integer (not strings).
// Code noErrors = 0 is indication that script finished without errors.
errorCodes: {
noErrors: 0,
badArguments: 1,
badGettingFileList: 2
}
};
// - - - - -
// thisOptions
// Purpose of thisOptions: organize set of hard-coded
// normalisation settings
// into a compact object package
var thisOptions = {
normNameBefore: 'input_',
normNameAfter: ''
}
// - - - - -
// outputToolbox depends on
// basicOptions && WScript
// Purpose of outputToolbox: organize in one group
// functions that are used to
// output chars or messages to stdout or stderr
var outputToolbox = {
// Syntactic sugar for printing exact set of chars to stdout.
print: function( data ) {
// For this script to output UTF characters correctly, program must
// run under /U switch - like
// cscript /U //NoLogo program-name-here.js arg1 arg2
WScript.StdOut.Write( data );
},
// Syntactic sugar for printing one line to stdout.
// terminated with your preferred line-terminator.
printLine: function( data ) {
outputToolbox.print( data );
outputToolbox.print( basicOptions.eolStdOut );
},
// Syntactic sugar for printing exact set of chars to stderr.
printErr: function( data ) {
WScript.StdErr.Write( data );
},
// Syntactic sugar for printing one line to stderr
// terminated with your preferred line-terminator.
printErrLine: function( data ) {
outputToolbox.printErr( data );
outputToolbox.printErr( basicOptions.eolStdErr );
},
// Syntactic sugar for printing one line to stderr
// prepended with meaningful script name (that is useful for debug)
// terminated with your preferred line-terminator.
printErrMessage: function( data ) {
outputToolbox.printErr( basicOptions.scriptName + ' failed' );
outputToolbox.printErr( data );
outputToolbox.printErr( basicOptions.eolStdErr );
},
// Syntactic sugar for printing one line to stderr
// prepended with meaningful script name (that is useful for debug)
// terminated with your preferred line-terminator.
printErrWarning: function( data ) {
outputToolbox.printErr( 'Warning for ' + basicOptions.scriptName + ': ' );
outputToolbox.printErr( data );
outputToolbox.printErr( basicOptions.eolStdErr );
},
// Syntactic sugar for printing Error objects
// catched by throw...catch construction
printErrCatchedErr: function( e ) {
outputToolbox.printErrMessage(
' - catched error details - ' + e.name + ': ' + e.message
);
}
};
// - - - - -
// flowToolbox depends on
// WScript
// Purpose of flowToolbox: organize in one group
// functions that are used to
// control general flow of program
var flowToolbox = {
// Shortcut to kill this script.
// When invoked it stops execution.
die: function ( code ) {
if ( typeof code === 'undefined' ) {
code = 0;
}
WScript.Quit(code);
}
};
// - - - - -
// Here is a cleaner alternative: we pollute only name $b.
// Not so easy to type but later, you can easily find all code calls
// dependant on functions from 'basis'.
// '$b' stands for 'library-b' or more exactly - "library named 'basis'"
var $b = {};
$b.print = outputToolbox.print;
$b.printLine = outputToolbox.printLine;
$b.printErr = outputToolbox.printErr;
$b.printErrLine = outputToolbox.printErrLine;
$b.printErrMessage = outputToolbox.printErrMessage;
$b.printErrWarning = outputToolbox.printErrWarning;
$b.die = flowToolbox.die;
// ^ You can use those as functions now
// - - - - -
// $fileListToolbox
// depends on WScript && ActiveXObject( "Scripting.FileSystemObject" ) && $b
var $fileListToolbox = {
/*
* Input:
* any gibberish that is acceptable
* for your local version of
* Scripting.FileSystemObject
* - like
* getFileListForFolder( 'C:\' );
* getFileListForFolder( 'C:\abc' );
* getFileListForFolder( './' );
* getFileListForFolder( './abc/../abc/..' );
*
* Output:
* aFileList with format:
* empty array
* or
* array of oFileData
* oFileData has format:
* {
* fileNameBase: ...[string],
* fileNameExtension: ...[string],
* parentAbsolutePath: ...[string]
* }
*/
getFileListForFolder: function( folderSpec )
{
var aResult = [];
var fso = new ActiveXObject( "Scripting.FileSystemObject" );
if ( !fso.FolderExists( folderSpec ) ) {
$b.printErrMessage(' folder="' + folderSpec + '" doesn\'t exist');
$b.die( basicOptions.errorCodes.badGettingFileList );
}
var folder = fso.GetFolder( folderSpec );
var files = new Enumerator( folder.files );
for ( ; !files.atEnd(); files.moveNext() ) {
var thisFileItem = files.item();
aResult.push({
fileNameBase: fso.GetBaseName( thisFileItem ),
fileNameExtension: fso.GetExtensionName( thisFileItem ),
parentAbsolutePath: (
fso.GetParentFolderName(
fso.GetAbsolutePathName( thisFileItem )
)
)
});
}
return( aResult );
},
/*
* Purpose:
* Sort files by fileNameBase
*
* Input:
* aFileList
* - format is the same
* as for function $fileListToolbox.getFileListForFolder
*
* Output:
* - format is the same
* as for function $fileListToolbox.getFileListForFolder
*/
sortFileListByFileNameBaseAsc: function( aFileList ) {
var fSort = function( a, b ) {
if ( a.fileNameBase > b.fileNameBase ) {
return 1;
}
if ( a.fileNameBase < b.fileNameBase ) {
return -1;
}
return 0;
}
return aFileList.sort( fSort );
},
/*
* Purpose:
* Tool for displaying contents of fileList
*
* Input:
* aFileList
* - format is the same
* as for function $fileListToolbox.getFileListForFolder
*
* Sideeffects:
* Prints aFileList to the stdout stream
*/
printFileList: function( aFileList, callbackPrintLine ) {
if ( typeof callbackPrintLine === 'undefined' ) {
return;
}
var i;
var L = aFileList.length;
var elem;
for ( i = 0; i < L; i++ ) {
elem = aFileList[i];
callbackPrintLine();
callbackPrintLine( 'i: ' + i );
callbackPrintLine( 'fileNameBase: ' + elem.fileNameBase );
callbackPrintLine( 'fileNameExtension: ' + elem.fileNameExtension );
callbackPrintLine( 'parentAbsolutePath: ' + elem.parentAbsolutePath );
}
}
};
/*
// basic test:
$fileListToolbox.printFileList(
$fileListToolbox.sortFileListByFileNameBaseAsc(
$fileListToolbox.getFileListForFolder( 'bTmp' )
),
$b.printLine
);
*/
// - - - - -
// $mappingEngine
var $mappingEngine = {
// this is a relic from a more complex fileformat...
printHeader: function(
functionPrintChars
) {
},
printBasicFilePair: function(
functionPrintChars,
aFileNameBase, aFileNameExtension,
bFileNameBase, bFileNameExtension
) {
var eol = '\r\n';
if ( aFileNameBase.match( /[\r\n]/ ) !== null ) {
throw new Error(
'error: bad input: aFileNameBase contains end of line symbols'
);
}
if ( bFileNameBase.match( /[\r\n]/ ) !== null ) {
throw new Error(
'error: bad input: bFileNameBase contains end of line symbols'
);
}
if ( aFileNameExtension.match( /[\r\n]/ ) !== null ) {
throw new Error(
'error: bad input: aFileNameExtension contains end of line symbols'
);
}
if ( bFileNameExtension.match( /[\r\n]/ ) !== null ) {
throw new Error(
'error: bad input: bFileNameExtension contains end of line symbols'
);
}
var ar = [
'id:1',
'baseName:' + aFileNameBase,
'extensionName:' + aFileNameExtension,
'id:2',
'baseName:' + bFileNameBase,
'extensionName:' + bFileNameExtension,
':',
''
];
functionPrintChars( ar.join( eol ) );
}
};
/*
//basic test:
$mappingEngine.printHeader( $b.print );
$mappingEngine.printBasicFilePair( $b.print, 'abcdef', 'ext', 'input_001', 'txt' );
$mappingEngine.printBasicFilePair( $b.print, 'ghijkl', 'jpg', 'input_002', 'png' );
*/
// it should print to output:
/*
id:1
baseName:abcdef
extensionName:ext
id:2
baseName:input_001
extensionName:txt
:
id:1
baseName:ghijkl
extensionName:jpg
id:2
baseName:input_002
extensionName:png
:
*/
// - - - - -
function generateFAddLeadingZeroes( minimumLength, startFrom ) {
if ( typeof minimumLength === 'undefined' ) {
minimumLength = 0;
}
if ( typeof startFrom === 'undefined' ) {
startFrom = 0;
}
minimumLength = parseInt( minimumLength ) - 1;
startFrom = parseInt( startFrom );
var fAddLeadingZeroes = function( i, L ) {
i += startFrom;
L += startFrom - 1;
var sResult = i.toString();
if ( minimumLength > 0 ) {
var level = 0;
var tmpL = L;
while (
( tmpL = parseInt( tmpL / 10 ) ) >= 1
) {
++level;
}
level = Math.max( level, minimumLength );
var iLength = sResult.length;
while ( iLength <= level-- ) {
sResult = '0' + sResult;
}
}
return sResult;
}
return fAddLeadingZeroes;
}
// - - - - -
var $thisArguments = {
getArgumentsObj: function() {
return WScript.Arguments;
// mockup for tests in browser
/*
var testArr = [1,2,3,4,5];
var mockupArguments = function( i ) {
return testArr[i];
};
mockupArguments.length = testArr.length;
return mockupArguments;
*/
},
printErrorMessageAndDie: function( meaningfulPart ) {
$b.printErrMessage(
' - problem when ' +
'parsing command-line arguments (options): ' +
''
);
$b.die( basicOptions.errorCodes.badArguments );
},
getArgumentsDummy: function() {
return {
folder: 'C:\abc',
minNOfDigits: 0,
countStart: 1
};
},
print: function( argumentsObject ) {
$b.printErrLine( 'folder: ' + argumentsObject.folder );
$b.printErrLine( 'minNOfDigits: ' + argumentsObject.minNOfDigits );
$b.printErrLine( 'countStart: ' + argumentsObject.countStart );
},
// We don't use any flags it this script...
/*
isThereAFlagInArguments: function( inString ) {
var normalizedEthalon = inString.toString().toLowerCase();
var objArgs = $thisArguments.getArgumentsObj();
var i;
var L = objArgs.length;
for ( i = 2; i < L; i++ ) {
var normalizedArgument = objArgs( i ).toString().toLowerCase();
if ( normalizedEthalon === normalizedArgument ) {
return true;
}
}
return false;
},
*/
getArguments: {
folder: function() {
var rR = '';
var objArgs = $thisArguments.getArgumentsObj();
if ( objArgs.length < 1 ) {
$b.printErrMessage(
' - reason: not enough arguments, ' +
'you MUST to provide at least 1 argument, ' +
'but ammount of arguments you provided ' +
'only: ' + objArgs.length
);
$b.die( basicOptions.errorCodes.badArguments );
}
rR = objArgs(0);
var fso = new ActiveXObject("Scripting.FileSystemObject");
rR = fso.GetAbsolutePathName( rR );
return rR;
},
minNOfDigits: function() {
var rR = '0';
var objArgs = $thisArguments.getArgumentsObj();
if ( objArgs.length >= 2 ) {
rR = objArgs(1);
}
var intRR = parseInt( rR );
// test for NaN
if ( intRR !== intRR ) {
$b.printErrMessage(
' - reason: bad argument 2, ' +
'you MUST to provide non-negative integer, ' +
'but you provided non-integer: ' +
'[' + ( typeof rR ) + ']' +
' "' + rR + '"' +
''
);
$b.die( basicOptions.errorCodes.badArguments );
}
// test for negative
if ( intRR < 0 ) {
$b.printErrMessage(
' - reason: bad argument 2, ' +
'you MUST to provide non-negative integer, ' +
'but you provided negtive integer: ' +
'[' + ( typeof rR ) + ']' +
' "' + rR + '"' +
''
);
$b.die( basicOptions.errorCodes.badArguments );
}
rR = intRR;
return rR;
},
countStart: function() {
var rR = '1';
var objArgs = $thisArguments.getArgumentsObj();
if ( objArgs.length >= 3 ) {
rR = objArgs(2);
}
var intRR = parseInt( rR );
// test for NaN
if ( intRR !== intRR ) {
$b.printErrMessage(
' - reason: bad argument 3, ' +
'you MUST to provide non-negative integer, ' +
'but you provided non-integer: ' +
'[' + ( typeof rR ) + ']' +
' "' + rR + '"' +
''
);
$b.die( basicOptions.errorCodes.badArguments );
}
// test for negative
if ( intRR < 0 ) {
$b.printErrMessage(
' - reason: bad argument 3, ' +
'you MUST to provide non-negative integer, ' +
'but you provided negtive integer: ' +
'[' + ( typeof rR ) + ']' +
' "' + rR + '"' +
''
);
$b.die( basicOptions.errorCodes.badArguments );
}
rR = intRR;
return rR;
}
},
getArgumentsAll: function() {
var $get = $thisArguments.getArguments;
var rR = {
folder: $get.folder(),
minNOfDigits: $get.minNOfDigits(),
countStart: $get.countStart()
};
return rR;
}
};
var $a = $thisArguments;
// - - - - -
function main() {
var $inputOptions = $a.getArgumentsAll();
$b.printErrLine();
$b.printErrLine(
'Arguments for ' + basicOptions.scriptName +
' are interpreted this way:'
);
$a.print( $inputOptions );
$b.printErrLine();
try {
var aFileList = $fileListToolbox.getFileListForFolder( $inputOptions.folder );
} catch (e) {
$b.printErrMessage(
' - problems while trying ' +
'to get list of files from folder: ' +
$inputOptions.folder
);
$b.die( basicOptions.errorCodes.badGettingFileList );
return;
}
aFileList = $fileListToolbox.sortFileListByFileNameBaseAsc( aFileList );
// addLeadingZeroes
// is a function now.
// You can see what arguments it requires
// and output it produces in source for
// generateFAddLeadingZeroes
// function.
var addLeadingZeroes = generateFAddLeadingZeroes(
$inputOptions.minNOfDigits,
$inputOptions.countStart
);
var normNameBefore = thisOptions.normNameBefore;
var normNameAfter = thisOptions.normNameAfter;
var i;
var L = aFileList.length;
for ( i = 0; i < L; i++ ) {
var elem = aFileList[i];
var aBase = elem.fileNameBase;
var aExtension = elem.fileNameExtension;
var count = addLeadingZeroes( i, L );
var bBase = normNameBefore + count + normNameAfter;
var bExtension = aExtension;
$mappingEngine.printBasicFilePair(
$b.print,
aBase, aExtension,
bBase, bExtension
);
}
$b.printErr( 'Script ' + basicOptions.scriptName + ' - finished!' );
};
main();

Listing for program map-filenames.js :
// - - - - -
// #magicVars #config
// Purpose of basicOptions: organize set of most basic settings
// into a compact object package
var basicOptions = {
// Fist things first -
// "scriptName" will be important for error messages.
// Because when you debug a pipeline of scripts, it is always
// crucial to know - which program in pipeline of 10 utilities
// throws that cryptic stderr message "error: not enough arguments".
// I mean huh? Was that calc-tool.js? Or sort-tool.js? Go figure...
scriptName: WScript.ScriptName.toString(),
// What kind of End Of Line sequence
// (they call it New Line sometimes too)
// you prefer when using outputToolbox?
eolStdOut: '\r\n',
eolStdErr: '\r\n',
// Those are all possible errorCodes that this script
// was designed to fail with.
// You can define other error codes.
// Those SHOULD be integer (not strings).
// Code noErrors = 0 is indication that script finished without errors.
errorCodes: {
noErrors: 0,
badArguments: 1,
badCopy: 2,
badParsing: 3,
badGettingFileList: 4
}
};
// - - - - -
// - - - - -
// outputToolbox depends on
// basicOptions && WScript
// Purpose of outputToolbox: organize in one group
// functions that are used to
// output chars or messages to stdout or stderr
var outputToolbox = {
// Syntactic sugar for printing exact set of chars to stdout.
print: function( data ) {
// For this script to output UTF characters correctly, program must
// run under /U switch - like
// cscript /U //NoLogo program-name-here.js arg1 arg2
WScript.StdOut.Write( data );
},
// Syntactic sugar for printing one line to stdout.
// terminated with your preferred line-terminator.
printLine: function( data ) {
outputToolbox.print( data );
outputToolbox.print( basicOptions.eolStdOut );
},
// Syntactic sugar for printing exact set of chars to stderr.
printErr: function( data ) {
WScript.StdErr.Write( data );
},
// Syntactic sugar for printing one line to stderr
// terminated with your preferred line-terminator.
printErrLine: function( data ) {
outputToolbox.printErr( data );
outputToolbox.printErr( basicOptions.eolStdErr );
},
// Syntactic sugar for printing one line to stderr
// prepended with meaningful script name (that is useful for debug)
// terminated with your preferred line-terminator.
printErrMessage: function( data ) {
outputToolbox.printErr( basicOptions.scriptName + ' failed' );
outputToolbox.printErr( data );
outputToolbox.printErr( basicOptions.eolStdErr );
},
// Syntactic sugar for printing one line to stderr
// prepended with meaningful script name (that is useful for debug)
// terminated with your preferred line-terminator.
printErrWarning: function( data ) {
outputToolbox.printErr( 'Warning for ' + basicOptions.scriptName + ': ' );
outputToolbox.printErr( data );
outputToolbox.printErr( basicOptions.eolStdErr );
},
// Syntactic sugar for printing Error objects
// catched by throw...catch construction
printErrCatchedErr: function( e ) {
outputToolbox.printErrMessage(
' - catched error details - ' + e.name + ': ' + e.message
);
}
};
// - - - - -
// flowToolbox depends on
// WScript
// Purpose of flowToolbox: organize in one group
// functions that are used to
// control general flow of program
var flowToolbox = {
// Shortcut to kill this script.
// When invoked it stops execution.
die: function ( code ) {
if ( typeof code === 'undefined' ) {
code = 0;
}
WScript.Quit(code);
}
};
// - - - - -
// Here is a cleaner alternative: we pollute only name $b.
// Not so easy to type but later, you can easily find all code calls
// dependant on functions from 'basis'.
// '$b' stands for 'library-b' or more exactly - "library named 'basis'"
var $b = {};
$b.print = outputToolbox.print;
$b.printLine = outputToolbox.printLine;
$b.printErr = outputToolbox.printErr;
$b.printErrLine = outputToolbox.printErrLine;
$b.printErrMessage = outputToolbox.printErrMessage;
$b.printErrWarning = outputToolbox.printErrWarning;
$b.die = flowToolbox.die;
// ^ You can use those as functions now
// - - - - -
// $fileListToolbox
// depends on WScript && ActiveXObject( "Scripting.FileSystemObject" ) && $b
var $fileListToolbox = {
/*
* Input:
* any gibberish that is acceptable
* for your local version of
* Scripting.FileSystemObject
* - like
* getFileListForFolder( 'C:\' );
* getFileListForFolder( 'C:\abc' );
* getFileListForFolder( './' );
* getFileListForFolder( './abc/../abc/..' );
*
* Output:
* aFileList with format:
* empty array
* or
* array of oFileData
* oFileData has format:
* {
* fileNameBase: ...[string],
* fileNameExtension: ...[string],
* parentAbsolutePath: ...[string]
* }
*/
getFileListForFolder: function( folderSpec )
{
var aResult = [];
var fso = new ActiveXObject( "Scripting.FileSystemObject" );
if ( !fso.FolderExists( folderSpec ) ) {
$b.printErrMessage(' folder="' + folderSpec + '" doesn\'t exist');
$b.die( basicOptions.errorCodes.badGettingFileList );
}
var folder = fso.GetFolder( folderSpec );
var files = new Enumerator( folder.files );
for ( ; !files.atEnd(); files.moveNext() ) {
var thisFileItem = files.item();
aResult.push({
fileNameBase: fso.GetBaseName( thisFileItem ),
fileNameExtension: fso.GetExtensionName( thisFileItem ),
parentAbsolutePath: (
fso.GetParentFolderName(
fso.GetAbsolutePathName( thisFileItem )
)
)
});
}
return( aResult );
},
/*
* Purpose:
* Sort files by fileNameBase
*
* Input:
* aFileList
* - format is the same
* as for function $fileListToolbox.getFileListForFolder
*
* Output:
* - format is the same
* as for function $fileListToolbox.getFileListForFolder
*/
sortFileListByFileNameBaseAsc: function( aFileList ) {
var fSort = function( a, b ) {
if ( a.fileNameBase > b.fileNameBase ) {
return 1;
}
if ( a.fileNameBase < b.fileNameBase ) {
return -1;
}
return 0;
}
return aFileList.sort( fSort );
},
/*
Output:
false
when no results
oFileData
when result found
*/
getAnyFileWithBaseName: function( baseName, aFileList ) {
var i;
var L = aFileList.length;
for ( i = 0; i < L; i++ ) {
var a = aFileList[i];
if ( a.fileNameBase === baseName ) {
return a;
}
}
return false;
},
/*
* Purpose:
* Tool for displaying contents of fileList
*
* Input:
* aFileList
* - format is the same
* as for function $fileListToolbox.getFileListForFolder
*
* Sideeffects:
* Prints aFileList to the stdout stream
*/
printFileList: function( aFileList, callbackPrintLine ) {
if ( typeof callbackPrintLine === 'undefined' ) {
return;
}
var i;
var L = aFileList.length;
var elem;
for ( i = 0; i < L; i++ ) {
elem = aFileList[i];
callbackPrintLine();
callbackPrintLine( 'i: ' + i );
callbackPrintLine( 'fileNameBase: ' + elem.fileNameBase );
callbackPrintLine( 'fileNameExtension: ' + elem.fileNameExtension );
callbackPrintLine( 'parentAbsolutePath: ' + elem.parentAbsolutePath );
}
}
};
/*
// basic test:
$fileListToolbox.printFileList(
$fileListToolbox.sortFileListByFileNameBaseAsc(
$fileListToolbox.getFileListForFolder( 'bTmp' )
),
$b.printLine
);
*/
/*
// test for fuzzy search only by file base name:
$fileListToolbox.printFileList(
$fileListToolbox.sortFileListByFileNameBaseAsc(
$fileListToolbox.getFileListForFolder( 'bTmp' )
),
$b.printLine
);
var fuzzy = $fileListToolbox.getAnyFileWithBaseName(
'input_1',
$fileListToolbox.getFileListForFolder( 'bTmp' )
);
$b.printLine( 'fuzzy.fileNameBase: ' + fuzzy.fileNameBase );
$b.printLine( 'fuzzy.fileNameExtension: ' + fuzzy.fileNameExtension );
*/
// $streamToolbox
var $streamToolbox = {
// when end of line returns '' (aka empty string)
getChar: function() {
if ( WScript.StdIn.AtEndOfStream ) {
return '';
}
return WScript.StdIn.Read(1);
},
// tests show that for Windows 7 it understands terminators
// CRLF or LF-only
// when end of line returns '' (aka empty string)
getLine: function() {
if ( WScript.StdIn.AtEndOfStream ) {
return '';
}
return WScript.StdIn.ReadLine();
}
}
var $s = $streamToolbox;
// it could be used like that:
/*
var piece;
while ( ( piece = $s.getLine() ) !== '' ) {
$b.printLine( 'piece: ' + '>' + piece + '<' );
}
*/
// $thisArguments
var $thisArguments = {
getArgumentsObj: function() {
return WScript.Arguments;
// mockup for tests in browser
/*
var testArr = [1,2,3,4,5];
var mockupArguments = function( i ) {
return testArr[i];
};
mockupArguments.length = testArr.length;
return mockupArguments;
*/
},
printErrorMessageAndDie: function( meaningfulPart ) {
$b.printErrMessage(
' - problem when ' +
'parsing command-line arguments (options): ' +
''
);
$b.die( basicOptions.errorCodes.badArguments );
},
getArgumentsDummy: function() {
return {
direction: 'from-id-2-to-id-1',
resultExtension: 'auto',
folderFrom: 'C:\abc',
folderTo: 'D:\abc',
mode: 'training',
modeForExtensionFrom: 'strict'
};
},
print: function( argumentsObject ) {
$b.printErrLine( 'folderFrom: ' + argumentsObject.folderFrom );
$b.printErrLine( 'folderTo: ' + argumentsObject.folderTo );
$b.printErrLine( 'direction: ' + argumentsObject.direction );
$b.printErrLine( 'resultExtension: ' + argumentsObject.resultExtension );
$b.printErrLine( 'mode: ' + argumentsObject.mode );
$b.printErrLine( 'modeForExtensionFrom: ' + argumentsObject.modeForExtensionFrom );
},
isThereAFlagInArguments: function( inString ) {
var normalizedEthalon = inString.toString().toLowerCase();
var objArgs = $thisArguments.getArgumentsObj();
var i;
var L = objArgs.length;
for ( i = 2; i < L; i++ ) {
var normalizedArgument = objArgs( i ).toString().toLowerCase();
if ( normalizedEthalon === normalizedArgument ) {
return true;
}
}
return false;
},
getArguments: {
folderFrom: function() {
var rR = '';
var objArgs = $thisArguments.getArgumentsObj();
if ( objArgs.length < 2 ) {
$b.printErrMessage(
' - reason: not enough arguments, ' +
'you MUST to provide at least 2 arguments, ' +
'but ammount of arguments you provided ' +
'is only: ' + objArgs.length
);
$b.die( basicOptions.errorCodes.badArguments );
}
rR = objArgs(0);
var fso = new ActiveXObject("Scripting.FileSystemObject");
rR = fso.GetAbsolutePathName( rR );
return rR;
},
folderTo: function() {
var rR = '';
var objArgs = $thisArguments.getArgumentsObj();
if ( objArgs.length < 2 ) {
$b.printErrMessage(
' - reason: not enough arguments, ' +
'you MUST to provide at least 2 arguments, ' +
'but ammount of arguments you provided ' +
'is only: ' + objArgs.length
);
$b.die( basicOptions.errorCodes.badArguments );
}
rR = objArgs(1);
var fso = new ActiveXObject("Scripting.FileSystemObject");
rR = fso.GetAbsolutePathName( rR );
return rR;
},
direction: function() {
var rR = 'from-id-1-to-id-2';
var hasFlag = $thisArguments.isThereAFlagInArguments;
if ( hasFlag( '/MapFrom2To1' ) ) {
rR = 'from-id-2-to-id-1';
}
return rR;
},
resultExtension: function() {
var rR = 'auto';
var hasFlag = $thisArguments.isThereAFlagInArguments;
if ( hasFlag( '/ResultExtension:none' ) ) {
rR = 'none';
}
if ( hasFlag( '/ResultExtension:21' ) ) {
rR = '21';
}
if ( hasFlag( '/ResultExtension:12' ) ) {
rR = '12';
}
if ( hasFlag( '/ResultExtension:2' ) ) {
rR = '2';
}
if ( hasFlag( '/ResultExtension:1' ) ) {
rR = '1';
}
return rR;
},
mode: function() {
var rR = 'training';
var hasFlag = $thisArguments.isThereAFlagInArguments;
if ( hasFlag( '/NotTraining' ) ) {
rR = 'not-training';
}
return rR;
},
modeForExtensionFrom: function() {
var rR = 'strict';
var hasFlag = $thisArguments.isThereAFlagInArguments;
if ( hasFlag( '/FuzzyExtensionsFrom' ) ) {
rR = 'fuzzy';
}
return rR;
}
},
getArgumentsAll: function() {
var $get = $thisArguments.getArguments;
var rR = {
folderFrom: $get.folderFrom(),
folderTo: $get.folderTo(),
direction: $get.direction(),
resultExtension: $get.resultExtension(),
mode: $get.mode(),
modeForExtensionFrom: $get.modeForExtensionFrom()
};
return rR;
}
};
var $a = $thisArguments;
//you can use it like this:
/*
var $inputOptions = $a.getArgumentsAll()
$a.print( $inputOptions );
*/
// $parser
// mockup options for separate testing
/*
var basicOptions = {
errorCodes: {
badParsing: 3
}
};
var $b = {
printErrMessage: function( a ) { console.log(a); },
die: function() { throw new Error(); }
};
*/
var $parser = {};
$parser.generateDummyLineReader = function( textBlock ) {
var lines = textBlock.split( '\r\n' );
var i = -1;
var L = lines.length;
var f = function() {
i++;
if ( i >= L ) {
return '';
}
return lines[i];
}
return f;
}
$parser.generateDummyInputOptions = function() {
return {
direction: '1-to-2',
resultExtension: '12',
folderFrom: 'C:\abc',
folderTo: 'D:\abc'
}
}
$parser.generateDummyMappingLineReader = function() {
return $parser.generateDummyLineReader(
[
'id:1',
'baseName:abcdef',
'extensionName:ext',
'id:2',
'baseName:input_001',
'extensionName:txt',
':',
'id:1',
'baseName:ghijkl',
'extensionName:jpg',
'id:2',
'baseName:input_002',
'extensionName:png',
':'
].join('\r\n')
);
}
/*
Remark:
functionReadNextLine
must be iterator that returns '' (aka empty string)
when stream of lines ends
functionForEachBlockDo must have
those arguments:
- id1Base
- id1Ext
- id2Base
- id2Ext
- $inputOptions
- functionCopyFile
$inputOptions must have properties
(wherever it mentioned in this block of comments):
- direction
- resultExtension
- folderFrom
- folderTo
functionCopyFile must have these arguments
- folderFrom
- fileFrom
- folderTo
- fileTo
*/
$parser.parseLinesAndForEachBlockDo = function(
functionReadNextLine,
functionForEachBlockDo,
$inputOptions,
functionCopyFile
) {
var buffer = null;
var nOfLine = 0;
var states = [
'waitingForId1',
'waitingForId2',
'waitingForBase',
'waitingForExtension',
'waitingForEndOfGroup'
];
function printErrorMessageAndDie( meaningfulPart ) {
$b.printErrMessage(
' - problem when parsing line ' + nOfLine + ': ' +
'[' + typeof line + ']' +
'"' + line + '"' +
meaningfulPart +
''
);
$b.die( basicOptions.errorCodes.badParsing );
}
var state = 'waitingForId1';
var line;
while (
( line = functionReadNextLine() ) !== ''
) {
++nOfLine;
var arOrNull = line.match( /^([a-zA-Z0-9-]*):(.*)$/ );
if (
( arOrNull === null )
) {
printErrorMessageAndDie(
'every line in mapping must be either' +
'"key:value" or ":"'
);
}
var key = arOrNull[1];
var value = arOrNull[2];
if (
( value.match( '\r' ) !== null )
) {
printErrorMessageAndDie(
'if line is ' +
'"key:value" ' +
'then it MUST NOT contain \r (CR) symbols ' +
'exept in terminator'
);
}
if ( state === 'waitingForId1' ) {
if (
( key === 'id' )
&&
( value === '1' )
) {
buffer = {};
buffer.nowGroup = {
id: 1
};
state = 'waitingForBase';
continue;
} else {
printErrorMessageAndDie(
'but this line must be "id:1"'
);
}
}
if ( state === 'waitingForBase' ) {
if (
( key === 'baseName' )
) {
buffer.nowGroup.base = value;
state = 'waitingForExtension';
continue;
} else {
printErrorMessageAndDie(
'but this line must be "baseName:...anything-here..."'
);
}
}
if ( state === 'waitingForExtension' ) {
if (
( key === 'extensionName' )
) {
buffer.nowGroup.ext = value;
if ( buffer.nowGroup.id === 1 ) {
state = 'waitingForId2';
} else {
state = 'waitingForEndOfGroup';
}
continue;
} else {
printErrorMessageAndDie(
'but this line must be "extensionName:...anything-here..."'
);
}
}
if ( state === 'waitingForId2' ) {
if (
( key === 'id' )
&&
( value === '2' )
) {
buffer.group1 = {
base: buffer.nowGroup.base,
ext: buffer.nowGroup.ext
}
buffer.nowGroup = {
id: 2
};
state = 'waitingForBase';
continue;
} else {
printErrorMessageAndDie(
'but this line must be "id:2"'
);
}
}
if ( state === 'waitingForEndOfGroup' ) {
if (
( key === '' )
&&
( value === '' )
) {
buffer.group2 = {
base: buffer.nowGroup.base,
ext: buffer.nowGroup.ext
}
functionForEachBlockDo(
buffer.group1.base,
buffer.group1.ext,
buffer.group2.base,
buffer.group2.ext,
$inputOptions,
functionCopyFile
);
buffer = null;
state = 'waitingForId1';
continue;
} else {
printErrorMessageAndDie(
'but this line must be ":"'
);
}
}
}
if ( buffer !== null ) {
printErrorMessageAndDie(
'dangling state (not all properties group are defined) - ' + state
);
}
}
/*
// test
// can be tested in for browser with only $parser object
$parser.parseLinesAndForEachBlockDo(
$parser.generateDummyMappingLineReader(),
function( a, b, c, d ) {
console.log( 'a:', a, 'b:', b, 'c:', c, 'd:', d );
}
);
*/
/*
Remark:
$inputOptions must have properties:
- direction
- resultExtension
- folderFrom
- folderTo
*/
$parser.reactionOnGoodParsedBlock = function(
id1Base, id1Ext,
id2Base, id2Ext,
$inputOptions,
functionCopyFile
) {
function glueExtensionToFileName(
fileName, extension
) {
fileName = fileName.toString();
extension = extension.toString();
if ( extension.length > 0 ) {
extension = '.' + extension;
}
return fileName + extension;
}
var folderFrom = $inputOptions.folderFrom;
var folderTo = $inputOptions.folderTo;
if ( $inputOptions.modeForExtensionFrom === 'fuzzy' ) {
var tmpFromBase = id1Base;
if ( $inputOptions.direction == 'from-id-2-to-id-1' ) {
tmpFromBase = id2Base;
}
var fuzzyFile = $fileListToolbox.getAnyFileWithBaseName(
tmpFromBase,
$fileListToolbox.getFileListForFolder( folderFrom )
);
if ( fuzzyFile !== false ) {
if ( $inputOptions.direction == 'from-id-2-to-id-1' ) {
id2Ext = fuzzyFile.fileNameExtension;
} else {
id1Ext = fuzzyFile.fileNameExtension;
}
}
}
var fromSet = { base: id1Base, ext: id1Ext };
var toSet = { base: id2Base, ext: id2Ext };
if ( $inputOptions.direction == 'from-id-2-to-id-1' ) {
fromSet = { base: id2Base, ext: id2Ext };
toSet = { base: id1Base, ext: id1Ext };
}
if ( $inputOptions.resultExtension === '21' ) {
toSet.ext = glueExtensionToFileName( id2Ext, id1Ext );
}
if ( $inputOptions.resultExtension === '12' ) {
toSet.ext = glueExtensionToFileName( id1Ext, id2Ext );
}
if ( $inputOptions.resultExtension === '2' ) {
toSet.ext = id2Ext;
}
if ( $inputOptions.resultExtension === '1' ) {
toSet.ext = id1Ext;
}
if ( $inputOptions.resultExtension === 'none' ) {
toSet.ext = '';
}
var fileFrom = glueExtensionToFileName( fromSet.base, fromSet.ext );
var fileTo = glueExtensionToFileName( toSet.base, toSet.ext );
functionCopyFile(
folderFrom, fileFrom,
folderTo, fileTo
);
}
/*
// test
// can be tested in for browser with only $parser object
$parser.reactionOnGoodParsedBlock(
'originalA', 'extA',
'originalB', 'extB',
{
direction: 'from-id-2-to-id-1',
resultExtension: 'auto',
folderFrom: 'C:\abc',
folderTo: 'D:\abc'
},
function(
folderFrom, fileFrom,
folderTo, fileTo
) {
console.log(
'folderFrom:', folderFrom,
'fileFrom:', fileFrom,
'folderTo:', folderTo,
'fileTo:', fileTo
);
}
);
// expected result:
// folderFrom: C:abc fileFrom: originalA.extA folderTo: D:abc fileTo: originalB.extB
*/
var $copyFileToolbox = {
copyFileTraining: function(
folderFrom, fileFrom,
folderTo, fileTo
) {
var fso = new ActiveXObject( "Scripting.FileSystemObject" );
if ( !fso.FolderExists( folderFrom ) ) {
$b.printErrWarning('folderFrom="' + folderFrom + '" doesn\'t exist');
}
if ( !fso.FolderExists( folderTo ) ) {
$b.printErrWarning('folderTo="' + folderTo + '" doesn\'t exist');
}
var pathFrom = fso.BuildPath( folderFrom, fileFrom );
var pathTo = fso.BuildPath( folderTo, fileTo );
if ( !fso.FileExists( pathFrom ) ) {
$b.printErrLine();
$b.printErrWarning('no source file found, script looking for it at pathFrom="' + pathFrom + '" - this file doesn\'t exist');
}
if ( fso.FileExists( pathTo ) ) {
$b.printErrLine();
$b.printErrWarning('be advised - there was found a file at destination for copy command, and in "NotTraining" mode you will overwrite file at pathTo="' + pathTo + '"');
}
$b.printLine( '' );
$b.printLine( 'Training mode: you ask to copy file' );
$b.printLine( 'from: ' + pathFrom );
$b.printLine( '\\/' );
$b.printLine( 'to: ' + pathTo );
},
copyFile: function(
folderFrom, fileFrom,
folderTo, fileTo
) {
var fso = new ActiveXObject( "Scripting.FileSystemObject" );
if ( !fso.FolderExists( folderFrom ) ) {
$b.printErrMessage(' folderFrom="' + folderFrom + '" doesn\'t exist');
$b.die( basicOptions.errorCodes.badCopy );
}
if ( !fso.FolderExists( folderTo ) ) {
$b.printErrMessage (' folderTo="' + folderTo + '" doesn\'t exist');
$b.die( basicOptions.errorCodes.badCopy );
}
var pathFrom = fso.BuildPath( folderFrom, fileFrom );
var pathTo = fso.BuildPath( folderTo, fileTo );
if ( !fso.FileExists( pathFrom ) ) {
$b.printErrMessage(' no source file found, script looking for it at pathFrom="' + pathFrom + '" doesn\'t exist');
$b.die( basicOptions.errorCodes.badCopy );
}
fso.copyFile( pathFrom, pathTo, true );
}
}
var $c = $copyFileToolbox;
// you can use it as:
/*
$c.copyFileTraining(
'./', '123.txt',
'b', '345.txt'
);
*/
// you will have a lot of problems testing if filename is unicode
// only way that I found for Windows 7 is to pass filename as stdin
// from some file (perhaps any streaming from file works ok here...)
// and yes, test shows that it works...
function main() {
var $inputOptions = $a.getArgumentsAll();
$b.printErrLine();
$b.printErrLine(
'Arguments for ' + basicOptions.scriptName +
' are interpreted this way:'
);
$a.print( $inputOptions );
$b.printErrLine();
var fCopyFile = $c.copyFileTraining;
if ( $inputOptions.mode === 'not-training' ) {
fCopyFile = $c.copyFile;
}
/*
var fuzzy = $fileListToolbox.getAnyFileWithBaseName(
'input_1',
$fileListToolbox.getFileListForFolder( 'bTmp' )
);
*/
$parser.parseLinesAndForEachBlockDo(
$s.getLine,
$parser.reactionOnGoodParsedBlock,
$inputOptions,
fCopyFile
);
$b.printErr( 'Script ' + basicOptions.scriptName + ' - finished!' );
};
main();

Related

Variable is not modified in this loop - no-unmodified-loop-condition from ESLint

I have a project where ESLint throws this error from while loop. It does not make sence to me. Error says:
497:14 error 'from' is not modified in this loop no-unmodified-loop-condition
497:22 error 'to' is not modified in this loop no-unmodified-loop-condition
This is the code (look at while cycle):
mediaSettings: (state) => {
const timestamps = [];
const events = [];
state.media.forEach( medium => {
if( !medium.settings.dates ) return;
medium.settings.dates.forEach( dateRange => {
if( !dateRange.date_from || !dateRange.date_to ) return;
const from = new Date( dateRange.date_from );
const to = new Date( dateRange.date_to );
while ( from <= to ) {
if( timestamps.includes(from.getTime()) ) {
from.setDate( from.getDate() + 1 );
continue;
}
events.push({
date: new Date( from.getTime() ), // Need Date clone via new Date().
mediumId: medium.id,
});
from.setDate( from.getDate() + 1 );
};
});
});
return events;
}
What is that? Can somebody tell me plase how to fix it? It does not make sence. This is not an error.
I found clever solution on this page which disables ESLint for some block of code and also a specific ESLint rules. It looks like:
mediaSettings: (state) => {
const timestamps = [];
const events = [];
state.media.forEach( medium => {
if( !medium.settings.dates ) return;
medium.settings.dates.forEach( dateRange => {
if( !dateRange.date_from || !dateRange.date_to ) return;
const from = new Date( dateRange.date_from );
const to = new Date( dateRange.date_to );
/* eslint-disable no-unmodified-loop-condition */
while ( from <= to ) {
if( timestamps.includes(from.getTime()) ) {
from.setDate( from.getDate() + 1 );
continue;
}
events.push({
date: new Date( from.getTime() ), // Need Date clone via new Date().
mediumId: medium.id,
});
from.setDate( from.getDate() + 1 );
};
/* eslint-enable no-unmodified-loop-condition */
});
});
return events;
}
You wouldn't get this error if you are using reassignable variables like this:
...
let from = new Date(dateRange.date_from)
let to = new Date(dateRange.date_to)
while (from <= to) {
if (timestamps.includes(from.getTime())) {
from = from.setDate(from.getDate() + 1)
continue
}
events.push({
date: new Date(from.getTime()), // Need Date clone via new Date().
mediumId: medium.id,
})
from = from.setDate(from.getDate() + 1)
}
...

AdWord Script Export to BigQuery "Empty Response"

Utilizing the following AdWords Script to export to BigQuery, the BigQuery.Jobs.insert is causing the script to terminate due to "Empty response". Any reason the call is not getting a response?
var ACCOUNTS = ['xxx','xxx'];
var CONFIG = {
BIGQUERY_PROJECT_ID: 'xxx',
BIGQUERY_DATASET_ID: 'xxx',
// Truncate existing data, otherwise will append.
TRUNCATE_EXISTING_DATASET: true,
TRUNCATE_EXISTING_TABLES: true,
// Back up reports to Google Drive.
WRITE_DATA_TO_DRIVE: false,
// Folder to put all the intermediate files.
DRIVE_FOLDER: 'Adwords Big Query Test',
// Default date range over which statistics fields are retrieved.
DEFAULT_DATE_RANGE: '20140101,20140105',
// Lists of reports and fields to retrieve from AdWords.
REPORTS: [{NAME: 'KEYWORDS_PERFORMANCE_REPORT',
CONDITIONS: 'WHERE Impressions>0',
FIELDS: {'AccountDescriptiveName' : 'STRING',
'Date' : 'STRING',
'CampaignId' : 'STRING',
'CampaignName' : 'STRING',
'AdGroupId' : 'STRING',
'AdGroupName' : 'STRING',
'Id' : 'STRING',
'Criteria' : 'STRING',
'KeywordMatchType' : 'STRING',
'AdNetworkType1' : 'STRING',
'AdNetworkType2' : 'STRING',
'Device' : 'STRING',
'AveragePosition' : 'STRING',
'QualityScore' : 'STRING',
'CpcBid' : 'STRING',
'TopOfPageCpc' : 'STRING',
'Impressions' : 'STRING',
'Clicks' : 'STRING',
'ConvertedClicks' : 'STRING',
'Cost' : 'STRING',
'Conversions' : 'STRING'
}
}],
RECIPIENT_EMAILS: [
'xxx',
]
};
function main() {
createDataset();
for (var i = 0; i < CONFIG.REPORTS.length; i++) {
var reportConfig = CONFIG.REPORTS[i];
createTable(reportConfig);
}
folder = getDriveFolder();
// Get an account iterator.
var accountIterator = MccApp.accounts().withIds(ACCOUNTS).withLimit(10).get();
var jobIdMap = {};
while (accountIterator.hasNext()) {
// Get the current account.
var account = accountIterator.next();
// Select the child account.
MccApp.select(account);
// Run reports against child account.
var accountJobIds = processReports(folder, account.getCustomerId());
jobIdMap[account.getCustomerId()] = accountJobIds;
}
waitTillJobsComplete(jobIdMap);
sendEmail(jobIdMap);
}
function createDataset() {
if (datasetExists()) {
if (CONFIG.TRUNCATE_EXISTING_DATASET) {
BigQuery.Datasets.remove(CONFIG.BIGQUERY_PROJECT_ID,
CONFIG.BIGQUERY_DATASET_ID, {'deleteContents' : true});
Logger.log('Truncated dataset.');
} else {
Logger.log('Dataset %s already exists. Will not recreate.',
CONFIG.BIGQUERY_DATASET_ID);
return;
}
}
// Create new dataset.
var dataSet = BigQuery.newDataset();
dataSet.friendlyName = CONFIG.BIGQUERY_DATASET_ID;
dataSet.datasetReference = BigQuery.newDatasetReference();
dataSet.datasetReference.projectId = CONFIG.BIGQUERY_PROJECT_ID;
dataSet.datasetReference.datasetId = CONFIG.BIGQUERY_DATASET_ID;
dataSet = BigQuery.Datasets.insert(dataSet, CONFIG.BIGQUERY_PROJECT_ID);
Logger.log('Created dataset with id %s.', dataSet.id);
}
/**
* Checks if dataset already exists in project.
*
* #return {boolean} Returns true if dataset already exists.
*/
function datasetExists() {
// Get a list of all datasets in project.
var datasets = BigQuery.Datasets.list(CONFIG.BIGQUERY_PROJECT_ID);
var datasetExists = false;
// Iterate through each dataset and check for an id match.
if (datasets.datasets != null) {
for (var i = 0; i < datasets.datasets.length; i++) {
var dataset = datasets.datasets[i];
if (dataset.datasetReference.datasetId == CONFIG.BIGQUERY_DATASET_ID) {
datasetExists = true;
break;
}
}
}
return datasetExists;
}
function createTable(reportConfig) {
if (tableExists(reportConfig.NAME)) {
if (CONFIG.TRUNCATE_EXISTING_TABLES) {
BigQuery.Tables.remove(CONFIG.BIGQUERY_PROJECT_ID,
CONFIG.BIGQUERY_DATASET_ID, reportConfig.NAME);
Logger.log('Truncated dataset %s.', reportConfig.NAME);
} else {
Logger.log('Table %s already exists. Will not recreate.',
reportConfig.NAME);
return;
}
}
// Create new table.
var table = BigQuery.newTable();
var schema = BigQuery.newTableSchema();
var bigQueryFields = [];
// Add account column to table.
var accountFieldSchema = BigQuery.newTableFieldSchema();
accountFieldSchema.description = 'AccountId';
accountFieldSchema.name = 'AccountId';
accountFieldSchema.type = 'STRING';
bigQueryFields.push(accountFieldSchema);
// Add each field to table schema.
var fieldNames = Object.keys(reportConfig.FIELDS);
for (var i = 0; i < fieldNames.length; i++) {
var fieldName = fieldNames[i];
var bigQueryFieldSchema = BigQuery.newTableFieldSchema();
bigQueryFieldSchema.description = fieldName;
bigQueryFieldSchema.name = fieldName;
bigQueryFieldSchema.type = reportConfig.FIELDS[fieldName];
bigQueryFields.push(bigQueryFieldSchema);
}
schema.fields = bigQueryFields;
table.schema = schema;
table.friendlyName = reportConfig.NAME;
table.tableReference = BigQuery.newTableReference();
table.tableReference.datasetId = CONFIG.BIGQUERY_DATASET_ID;
table.tableReference.projectId = CONFIG.BIGQUERY_PROJECT_ID;
table.tableReference.tableId = reportConfig.NAME;
table = BigQuery.Tables.insert(table, CONFIG.BIGQUERY_PROJECT_ID,
CONFIG.BIGQUERY_DATASET_ID);
Logger.log('Created table with id %s.', table.id);
}
function tableExists(tableId) {
// Get a list of all tables in the dataset.
var tables = BigQuery.Tables.list(CONFIG.BIGQUERY_PROJECT_ID,
CONFIG.BIGQUERY_DATASET_ID);
var tableExists = false;
// Iterate through each table and check for an id match.
if (tables.tables != null) {
for (var i = 0; i < tables.tables.length; i++) {
var table = tables.tables[i];
if (table.tableReference.tableId == tableId) {
tableExists = true;
break;
}
}
}
return tableExists;
}
function processReports(folder, accountId) {
var jobIds = [];
// Iterate over each report type.
for (var i = 0; i < CONFIG.REPORTS.length; i++) {
var reportConfig = CONFIG.REPORTS[i];
Logger.log('Running report %s for account %s', reportConfig.NAME,
accountId);
// Get data as csv
var csvData = retrieveAdwordsReport(reportConfig, accountId);
// If configured, back up data.
if (CONFIG.WRITE_DATA_TO_DRIVE) {
var fileName = reportConfig.NAME + '_' + accountId;
folder.createFile(fileName, csvData, MimeType.CSV);
Logger.log('Exported data to Drive folder ' +
CONFIG.DRIVE_FOLDER + ' for report ' + fileName);
}
// Convert to Blob format.
var blobData = Utilities.newBlob(csvData, 'application/octet-stream');
// Load data
var jobId = loadDataToBigquery(reportConfig, blobData);
jobIds.push(jobId);
}
return jobIds;
}
function retrieveAdwordsReport(reportConfig, accountId) {
var fieldNames = Object.keys(reportConfig.FIELDS);
var report = AdWordsApp.report(
'SELECT ' + fieldNames.join(',') +
' FROM ' + reportConfig.NAME + ' ' + reportConfig.CONDITIONS +
' DURING ' + CONFIG.DEFAULT_DATE_RANGE);
var rows = report.rows();
var csvRows = [];
// Header row
csvRows.push('AccountId,'+fieldNames.join(','));
// Iterate over each row.
while (rows.hasNext()) {
var row = rows.next();
var csvRow = [];
csvRow.push(accountId);
for (var i = 0; i < fieldNames.length; i++) {
var fieldName = fieldNames[i];
var fieldValue = row[fieldName].toString();
var fieldType = reportConfig.FIELDS[fieldName];
/* Strip off % and perform any other formatting here.
if ((fieldType == 'FLOAT' || fieldType == 'INTEGER') &&
fieldValue.charAt(fieldValue.length - 1) == '%') {
fieldValue = fieldValue.substring(0, fieldValue.length - 1);
}*/
// Add double quotes to any string values.
if (fieldType == 'STRING') {
fieldValue = fieldValue.replace(',', ''); //Handle fields with comma in value returned
fieldValue = fieldValue.replace('"', ''); //Handle fields with double quotes in value returned
fieldValue = fieldValue.replace('+', ''); //Handle fields with "+" in value returned
fieldValue = '"' + fieldValue + '"';
}
csvRow.push(fieldValue);
}
csvRows.push(csvRow.join(','));
}
Logger.log('Downloaded ' + reportConfig.NAME + ' for account ' + accountId +
' with ' + csvRows.length + ' rows.');
return csvRows.join('\n');
}
function getDriveFolder() {
var folders = DriveApp.getFoldersByName(CONFIG.DRIVE_FOLDER);
// Assume first folder is the correct one.
if (folders.hasNext()) {
Logger.log('Folder name found. Using existing folder.');
return folders.next();
}
return DriveApp.createFolder(CONFIG.DRIVE_FOLDER);
}
function loadDataToBigquery(reportConfig, data) {
function guid() {
function s4() {
return Math.floor((1 + Math.random()) * 0x10000)
.toString(16)
.substring(1);
}
return s4() + s4() + s4() + s4() + s4() + s4() + s4() + s4();
}
var makeId = guid();
var job = {
jobReference: {
jobId: makeId
},
configuration: {
load: {
destinationTable: {
projectId: CONFIG.BIGQUERY_PROJECT_ID,
datasetId: CONFIG.BIGQUERY_DATASET_ID,
tableId: reportConfig.NAME
},
skipLeadingRows: 1,
ignoreUnknownValues: true,
allowJaggedRows: true,
allowLargeResults: true
}
}
};
var insertJob = BigQuery.Jobs.insert(job, CONFIG.BIGQUERY_PROJECT_ID, data);
Logger.log('Load job started for %s. Check on the status of it here: ' +
'https://bigquery.cloud.google.com/jobs/%s', reportConfig.NAME,
CONFIG.BIGQUERY_PROJECT_ID);
return job.jobReference.jobId;
}
function waitTillJobsComplete(jobIdMap) {
var complete = false;
var remainingJobs = [];
var accountIds = Object.keys(jobIdMap);
for (var i = 0; i < accountIds.length; i++){
var accountJobIds = jobIdMap[accountIds[i]];
remainingJobs.push.apply(remainingJobs, accountJobIds);
}
while (!complete) {
if (AdWordsApp.getExecutionInfo().getRemainingTime() < 5){
Logger.log('Script is about to timeout, jobs ' + remainingJobs.join(',') +
' are still incomplete.');
}
remainingJobs = getIncompleteJobs(remainingJobs);
if (remainingJobs.length == 0) {
complete = true;
}
if (!complete) {
Logger.log(remainingJobs.length + ' jobs still being processed.');
// Wait 5 seconds before checking status again.
Utilities.sleep(5000);
}
}
Logger.log('All jobs processed.');
}
function getIncompleteJobs(jobIds) {
var remainingJobIds = [];
for (var i = 0; i < jobIds.length; i++) {
var jobId = jobIds[i];
var getJob = BigQuery.Jobs.get(CONFIG.BIGQUERY_PROJECT_ID, jobId);
if (getJob.status.state != 'DONE') {
remainingJobIds.push(jobId);
}
}
return remainingJobIds;
}
It appears the "Empty Response" error is being thrown on:
var insertJob = BigQuery.Jobs.insert(job, CONFIG.BIGQUERY_PROJECT_ID, data);
Have tried quite a few tweaks, but the answer doesn't appear to obvious to me. Thanks for any help!
I can be wrong but - I think that problem was with jobId because of issue with guid() function - missing "+" sign.
function guid() {
function s4() {
return Math.floor((1 + Math.random()) * 0x10000)
.toString(16)
.substring(1);
}
return s4() + s4() + s4() + s4() + s4() s4() + s4() + s4();
}
Why not to use jobId from Response like below?
var job = {
configuration: {
load: {
destinationTable: {
projectId: CONFIG.BIGQUERY_PROJECT_ID,
datasetId: CONFIG.BIGQUERY_DATASET_ID,
tableId: reportConfig.NAME
},
skipLeadingRows: 1,
ignoreUnknownValues: true,
allowJaggedRows: true,
allowLargeResults: true
}
}
};
var insertJob = BigQuery.Jobs.insert(job, CONFIG.BIGQUERY_PROJECT_ID, data);
Logger.log('Load job started for %s. Check on the status of it here: ' +
'https://bigquery.cloud.google.com/jobs/%s', reportConfig.NAME,
CONFIG.BIGQUERY_PROJECT_ID);
return insertJob.jobReference.jobId;
Added
In this case I would suggest to log jobId (makeId = guid()) and get job status following below link https://cloud.google.com/bigquery/docs/reference/v2/jobs/get#try-it
Enter ProjectId and JobId and you at least will see what is going on with your job!!
AdWords places a "--" in for null values. If you define your report fields as anything but string (e.g., float, integer, etc.) the insert will fail because it can't convert the dash dash to a float or integer.
Try setting all of your fields to string and see if that solves the problem.
Have you tried setting the WRITE_DATA_TO_DRIVE parameter to true to confirm that the report export is successful? How large is the result? I get the same error when attempting an insert greater than 10MB (~25k rows depending on columns). If the file export to Google Drive looks good, you can add a condition to the while loop in retrieveAdwordsReport to limit the file size. There was also a post on https://groups.google.com/forum/#!forum/adwords-scripts mentioning an issue when including AdNetworkType columns: https://groups.google.com/forum/#!searchin/adwords-scripts/adnetworktype2%7Csort:relevance/adwords-scripts/yK57JHCt3Cw/Cl1SjFaQBQAJ.
Limit result size:
var processedRows = 0;
// Iterate over each row.
while (rows.hasNext() && ++processedRows < 5000) {
var row = rows.next();
var csvRow = [];
csvRow.push(accountId);
if (processedRows % 1000 == 0)
{
Logger.log('Processed %s rows.',processedRows);
}
...

Select all objects with font-size between two sizes in illustrator?

I need to select all text-objects with a size between two values, for example 12 and 14pt (including 12.1, 12.2 etc). Is this at all possible?
This seems to be the candidate for a script. Try this:
function selectTextWhosePointSizeIs ( minPointSize, maxPointSize )
{
var doc, tfs, i = 0, n = 0, selectionArray = [];
if ( !app.documents.length ) { return; }
doc = app.activeDocument;
tfs = doc.textFrames;
n = tfs.length;
if ( !n ){ return; }
if ( isNaN ( minPointSize ) )
{
alert(minPointSize + " is not a valid number" );
return;
}
else if ( isNaN ( maxPointSize ) )
{
alert(maxPointSize + " is not a valid number" );
return;
}
else if ( minPointSize > maxPointSize )
{
alert(minPointSize + " can't be greater than "+ maxPointSize);
return;
}
for ( i = 0 ; i < n ; i++ )
{
if ( tfs[i].textRange.size >= minPointSize && tfs[i].textRange.size <= maxPointSize )
{
selectionArray [ selectionArray.length ] = tfs[i];
}
}
if ( selectionArray.length )
{
app.selection = selectionArray;
}
else
{
alert("Nothing found in this range.");
}
}
selectTextWhosePointSizeIs ( 12, 14 );
Hope it helps,
Loic

Parsing Data in Silverlight [duplicate]

Where could I find some JavaScript code to parse CSV data?
You can use the CSVToArray() function mentioned in this blog entry.
<script type="text/javascript">
// ref: http://stackoverflow.com/a/1293163/2343
// This will parse a delimited string into an array of
// arrays. The default delimiter is the comma, but this
// can be overriden in the second argument.
function CSVToArray( strData, strDelimiter ){
// Check to see if the delimiter is defined. If not,
// then default to comma.
strDelimiter = (strDelimiter || ",");
// Create a regular expression to parse the CSV values.
var objPattern = new RegExp(
(
// Delimiters.
"(\\" + strDelimiter + "|\\r?\\n|\\r|^)" +
// Quoted fields.
"(?:\"([^\"]*(?:\"\"[^\"]*)*)\"|" +
// Standard fields.
"([^\"\\" + strDelimiter + "\\r\\n]*))"
),
"gi"
);
// Create an array to hold our data. Give the array
// a default empty first row.
var arrData = [[]];
// Create an array to hold our individual pattern
// matching groups.
var arrMatches = null;
// Keep looping over the regular expression matches
// until we can no longer find a match.
while (arrMatches = objPattern.exec( strData )){
// Get the delimiter that was found.
var strMatchedDelimiter = arrMatches[ 1 ];
// Check to see if the given delimiter has a length
// (is not the start of string) and if it matches
// field delimiter. If id does not, then we know
// that this delimiter is a row delimiter.
if (
strMatchedDelimiter.length &&
strMatchedDelimiter !== strDelimiter
){
// Since we have reached a new row of data,
// add an empty row to our data array.
arrData.push( [] );
}
var strMatchedValue;
// Now that we have our delimiter out of the way,
// let's check to see which kind of value we
// captured (quoted or unquoted).
if (arrMatches[ 2 ]){
// We found a quoted value. When we capture
// this value, unescape any double quotes.
strMatchedValue = arrMatches[ 2 ].replace(
new RegExp( "\"\"", "g" ),
"\""
);
} else {
// We found a non-quoted value.
strMatchedValue = arrMatches[ 3 ];
}
// Now that we have our value string, let's add
// it to the data array.
arrData[ arrData.length - 1 ].push( strMatchedValue );
}
// Return the parsed data.
return( arrData );
}
</script>
jQuery-CSV
It's a jQuery plugin designed to work as an end-to-end solution for parsing CSV into JavaScript data. It handles every single edge case presented in RFC 4180, as well as some that pop up for Excel/Google spreadsheet exports (i.e., mostly involving null values) that the specification is missing.
Example:
track,artist,album,year
Dangerous,'Busta Rhymes','When Disaster Strikes',1997
// Calling this
music = $.csv.toArrays(csv)
// Outputs...
[
["track", "artist", "album", "year"],
["Dangerous", "Busta Rhymes", "When Disaster Strikes", "1997"]
]
console.log(music[1][2]) // Outputs: 'When Disaster Strikes'
Update:
Oh yeah, I should also probably mention that it's completely configurable.
music = $.csv.toArrays(csv, {
delimiter: "'", // Sets a custom value delimiter character
separator: ';', // Sets a custom field separator character
});
Update 2:
It now works with jQuery on Node.js too. So you have the option of doing either client-side or server-side parsing with the same library.
Update 3:
Since the Google Code shutdown, jquery-csv has been migrated to GitHub.
Disclaimer: I am also the author of jQuery-CSV.
Here's an extremely simple CSV parser that handles quoted fields with commas, new lines, and escaped double quotation marks. There's no splitting or regular expression. It scans the input string 1-2 characters at a time and builds an array.
Test it at http://jsfiddle.net/vHKYH/.
function parseCSV(str) {
var arr = [];
var quote = false; // 'true' means we're inside a quoted field
// Iterate over each character, keep track of current row and column (of the returned array)
for (var row = 0, col = 0, c = 0; c < str.length; c++) {
var cc = str[c], nc = str[c+1]; // Current character, next character
arr[row] = arr[row] || []; // Create a new row if necessary
arr[row][col] = arr[row][col] || ''; // Create a new column (start with empty string) if necessary
// If the current character is a quotation mark, and we're inside a
// quoted field, and the next character is also a quotation mark,
// add a quotation mark to the current column and skip the next character
if (cc == '"' && quote && nc == '"') { arr[row][col] += cc; ++c; continue; }
// If it's just one quotation mark, begin/end quoted field
if (cc == '"') { quote = !quote; continue; }
// If it's a comma and we're not in a quoted field, move on to the next column
if (cc == ',' && !quote) { ++col; continue; }
// If it's a newline (CRLF) and we're not in a quoted field, skip the next character
// and move on to the next row and move to column 0 of that new row
if (cc == '\r' && nc == '\n' && !quote) { ++row; col = 0; ++c; continue; }
// If it's a newline (LF or CR) and we're not in a quoted field,
// move on to the next row and move to column 0 of that new row
if (cc == '\n' && !quote) { ++row; col = 0; continue; }
if (cc == '\r' && !quote) { ++row; col = 0; continue; }
// Otherwise, append the current character to the current column
arr[row][col] += cc;
}
return arr;
}
I have an implementation as part of a spreadsheet project.
This code is not yet tested thoroughly, but anyone is welcome to use it.
As some of the answers noted though, your implementation can be much simpler if you actually have DSV or TSV file, as they disallow the use of the record and field separators in the values. CSV, on the other hand, can actually have commas and newlines inside a field, which breaks most regular expression and split-based approaches.
var CSV = {
parse: function(csv, reviver) {
reviver = reviver || function(r, c, v) { return v; };
var chars = csv.split(''), c = 0, cc = chars.length, start, end, table = [], row;
while (c < cc) {
table.push(row = []);
while (c < cc && '\r' !== chars[c] && '\n' !== chars[c]) {
start = end = c;
if ('"' === chars[c]){
start = end = ++c;
while (c < cc) {
if ('"' === chars[c]) {
if ('"' !== chars[c+1]) {
break;
}
else {
chars[++c] = ''; // unescape ""
}
}
end = ++c;
}
if ('"' === chars[c]) {
++c;
}
while (c < cc && '\r' !== chars[c] && '\n' !== chars[c] && ',' !== chars[c]) {
++c;
}
} else {
while (c < cc && '\r' !== chars[c] && '\n' !== chars[c] && ',' !== chars[c]) {
end = ++c;
}
}
row.push(reviver(table.length-1, row.length, chars.slice(start, end).join('')));
if (',' === chars[c]) {
++c;
}
}
if ('\r' === chars[c]) {
++c;
}
if ('\n' === chars[c]) {
++c;
}
}
return table;
},
stringify: function(table, replacer) {
replacer = replacer || function(r, c, v) { return v; };
var csv = '', c, cc, r, rr = table.length, cell;
for (r = 0; r < rr; ++r) {
if (r) {
csv += '\r\n';
}
for (c = 0, cc = table[r].length; c < cc; ++c) {
if (c) {
csv += ',';
}
cell = replacer(r, c, table[r][c]);
if (/[,\r\n"]/.test(cell)) {
cell = '"' + cell.replace(/"/g, '""') + '"';
}
csv += (cell || 0 === cell) ? cell : '';
}
}
return csv;
}
};
csvToArray v1.3
A compact (645 bytes), but compliant function to convert a CSV string into a 2D array, conforming to the RFC4180 standard.
https://code.google.com/archive/p/csv-to-array/downloads
Common Usage: jQuery
$.ajax({
url: "test.csv",
dataType: 'text',
cache: false
}).done(function(csvAsString){
csvAsArray=csvAsString.csvToArray();
});
Common usage: JavaScript
csvAsArray = csvAsString.csvToArray();
Override field separator
csvAsArray = csvAsString.csvToArray("|");
Override record separator
csvAsArray = csvAsString.csvToArray("", "#");
Override Skip Header
csvAsArray = csvAsString.csvToArray("", "", 1);
Override all
csvAsArray = csvAsString.csvToArray("|", "#", 1);
Here's my PEG(.js) grammar that seems to do ok at RFC 4180 (i.e. it handles the examples at http://en.wikipedia.org/wiki/Comma-separated_values):
start
= [\n\r]* first:line rest:([\n\r]+ data:line { return data; })* [\n\r]* { rest.unshift(first); return rest; }
line
= first:field rest:("," text:field { return text; })*
& { return !!first || rest.length; } // ignore blank lines
{ rest.unshift(first); return rest; }
field
= '"' text:char* '"' { return text.join(''); }
/ text:[^\n\r,]* { return text.join(''); }
char
= '"' '"' { return '"'; }
/ [^"]
Try it out at http://jsfiddle.net/knvzk/10 or http://pegjs.majda.cz/online. Download the generated parser at https://gist.github.com/3362830.
Here's another solution. This uses:
a coarse global regular expression for splitting the CSV string (which includes surrounding quotes and trailing commas)
fine-grained regular expression for cleaning up the surrounding quotes and trailing commas
also, has type correction differentiating strings, numbers, boolean values and null values
For the following input string:
"This is\, a value",Hello,4,-123,3.1415,'This is also\, possible',true,
The code outputs:
[
"This is, a value",
"Hello",
4,
-123,
3.1415,
"This is also, possible",
true,
null
]
Here's my implementation of parseCSVLine() in a runnable code snippet:
function parseCSVLine(text) {
return text.match( /\s*(\"[^"]*\"|'[^']*'|[^,]*)\s*(,|$)/g ).map( function (text) {
let m;
if (m = text.match(/^\s*,?$/)) return null; // null value
if (m = text.match(/^\s*\"([^"]*)\"\s*,?$/)) return m[1]; // Double Quoted Text
if (m = text.match(/^\s*'([^']*)'\s*,?$/)) return m[1]; // Single Quoted Text
if (m = text.match(/^\s*(true|false)\s*,?$/)) return m[1] === "true"; // Boolean
if (m = text.match(/^\s*((?:\+|\-)?\d+)\s*,?$/)) return parseInt(m[1]); // Integer Number
if (m = text.match(/^\s*((?:\+|\-)?\d*\.\d*)\s*,?$/)) return parseFloat(m[1]); // Floating Number
if (m = text.match(/^\s*(.*?)\s*,?$/)) return m[1]; // Unquoted Text
return text;
} );
}
let data = `"This is\, a value",Hello,4,-123,3.1415,'This is also\, possible',true,`;
let obj = parseCSVLine(data);
console.log( JSON.stringify( obj, undefined, 2 ) );
Here's my simple vanilla JavaScript code:
let a = 'one,two,"three, but with a comma",four,"five, with ""quotes"" in it.."'
console.log(splitQuotes(a))
function splitQuotes(line) {
if(line.indexOf('"') < 0)
return line.split(',')
let result = [], cell = '', quote = false;
for(let i = 0; i < line.length; i++) {
char = line[i]
if(char == '"' && line[i+1] == '"') {
cell += char
i++
} else if(char == '"') {
quote = !quote;
} else if(!quote && char == ',') {
result.push(cell)
cell = ''
} else {
cell += char
}
if ( i == line.length-1 && cell) {
result.push(cell)
}
}
return result
}
I'm not sure why I couldn't get Kirtan's example to work for me. It seemed to be failing on empty fields or maybe fields with trailing commas...
This one seems to handle both.
I did not write the parser code, just a wrapper around the parser function to make this work for a file. See attribution.
var Strings = {
/**
* Wrapped CSV line parser
* #param s String delimited CSV string
* #param sep Separator override
* #attribution: http://www.greywyvern.com/?post=258 (comments closed on blog :( )
*/
parseCSV : function(s,sep) {
// http://stackoverflow.com/questions/1155678/javascript-string-newline-character
var universalNewline = /\r\n|\r|\n/g;
var a = s.split(universalNewline);
for(var i in a){
for (var f = a[i].split(sep = sep || ","), x = f.length - 1, tl; x >= 0; x--) {
if (f[x].replace(/"\s+$/, '"').charAt(f[x].length - 1) == '"') {
if ((tl = f[x].replace(/^\s+"/, '"')).length > 1 && tl.charAt(0) == '"') {
f[x] = f[x].replace(/^\s*"|"\s*$/g, '').replace(/""/g, '"');
} else if (x) {
f.splice(x - 1, 2, [f[x - 1], f[x]].join(sep));
} else f = f.shift().split(sep).concat(f);
} else f[x].replace(/""/g, '"');
} a[i] = f;
}
return a;
}
}
Regular expressions to the rescue! These few lines of code handle properly quoted fields with embedded commas, quotes, and newlines based on the RFC 4180 standard.
function parseCsv(data, fieldSep, newLine) {
fieldSep = fieldSep || ',';
newLine = newLine || '\n';
var nSep = '\x1D';
var qSep = '\x1E';
var cSep = '\x1F';
var nSepRe = new RegExp(nSep, 'g');
var qSepRe = new RegExp(qSep, 'g');
var cSepRe = new RegExp(cSep, 'g');
var fieldRe = new RegExp('(?<=(^|[' + fieldSep + '\\n]))"(|[\\s\\S]+?(?<![^"]"))"(?=($|[' + fieldSep + '\\n]))', 'g');
var grid = [];
data.replace(/\r/g, '').replace(/\n+$/, '').replace(fieldRe, function(match, p1, p2) {
return p2.replace(/\n/g, nSep).replace(/""/g, qSep).replace(/,/g, cSep);
}).split(/\n/).forEach(function(line) {
var row = line.split(fieldSep).map(function(cell) {
return cell.replace(nSepRe, newLine).replace(qSepRe, '"').replace(cSepRe, ',');
});
grid.push(row);
});
return grid;
}
const csv = 'A1,B1,C1\n"A ""2""","B, 2","C\n2"';
const separator = ','; // field separator, default: ','
const newline = ' <br /> '; // newline representation in case a field contains newlines, default: '\n'
var grid = parseCsv(csv, separator, newline);
// expected: [ [ 'A1', 'B1', 'C1' ], [ 'A "2"', 'B, 2', 'C <br /> 2' ] ]
You don't need a parser-generator such as lex/yacc. The regular expression handles RFC 4180 properly thanks to positive lookbehind, negative lookbehind, and positive lookahead.
Clone/download code at https://github.com/peterthoeny/parse-csv-js
Just throwing this out there.. I recently ran into the need to parse CSV columns with Javascript, and I opted for my own simple solution. It works for my needs, and may help someone else.
const csvString = '"Some text, some text",,"",true,false,"more text","more,text, more, text ",true';
const parseCSV = text => {
const lines = text.split('\n');
const output = [];
lines.forEach(line => {
line = line.trim();
if (line.length === 0) return;
const skipIndexes = {};
const columns = line.split(',');
output.push(columns.reduce((result, item, index) => {
if (skipIndexes[index]) return result;
if (item.startsWith('"') && !item.endsWith('"')) {
while (!columns[index + 1].endsWith('"')) {
index++;
item += `,${columns[index]}`;
skipIndexes[index] = true;
}
index++;
skipIndexes[index] = true;
item += `,${columns[index]}`;
}
result.push(item);
return result;
}, []));
});
return output;
};
console.log(parseCSV(csvString));
Personally I like to use deno std library since most modules are officially compatible with the browser
The problem is that the std is in typescript but official solution might happen in the future https://github.com/denoland/deno_std/issues/641 https://github.com/denoland/dotland/issues/1728
For now there is an actively maintained on the fly transpiler https://bundle.deno.dev/
so you can use it simply like this
<script type="module">
import { parse } from "https://bundle.deno.dev/https://deno.land/std#0.126.0/encoding/csv.ts"
console.log(await parse("a,b,c\n1,2,3"))
</script>
I have constructed this JavaScript script to parse a CSV in string to array object. I find it better to break down the whole CSV into lines, fields and process them accordingly. I think that it will make it easy for you to change the code to suit your need.
//
//
// CSV to object
//
//
const new_line_char = '\n';
const field_separator_char = ',';
function parse_csv(csv_str) {
var result = [];
let line_end_index_moved = false;
let line_start_index = 0;
let line_end_index = 0;
let csr_index = 0;
let cursor_val = csv_str[csr_index];
let found_new_line_char = get_new_line_char(csv_str);
let in_quote = false;
// Handle \r\n
if (found_new_line_char == '\r\n') {
csv_str = csv_str.split(found_new_line_char).join(new_line_char);
}
// Handle the last character is not \n
if (csv_str[csv_str.length - 1] !== new_line_char) {
csv_str += new_line_char;
}
while (csr_index < csv_str.length) {
if (cursor_val === '"') {
in_quote = !in_quote;
} else if (cursor_val === new_line_char) {
if (in_quote === false) {
if (line_end_index_moved && (line_start_index <= line_end_index)) {
result.push(parse_csv_line(csv_str.substring(line_start_index, line_end_index)));
line_start_index = csr_index + 1;
} // Else: just ignore line_end_index has not moved or line has not been sliced for parsing the line
} // Else: just ignore because we are in a quote
}
csr_index++;
cursor_val = csv_str[csr_index];
line_end_index = csr_index;
line_end_index_moved = true;
}
// Handle \r\n
if (found_new_line_char == '\r\n') {
let new_result = [];
let curr_row;
for (var i = 0; i < result.length; i++) {
curr_row = [];
for (var j = 0; j < result[i].length; j++) {
curr_row.push(result[i][j].split(new_line_char).join('\r\n'));
}
new_result.push(curr_row);
}
result = new_result;
}
return result;
}
function parse_csv_line(csv_line_str) {
var result = [];
//let field_end_index_moved = false;
let field_start_index = 0;
let field_end_index = 0;
let csr_index = 0;
let cursor_val = csv_line_str[csr_index];
let in_quote = false;
// Pretend that the last char is the separator_char to complete the loop
csv_line_str += field_separator_char;
while (csr_index < csv_line_str.length) {
if (cursor_val === '"') {
in_quote = !in_quote;
} else if (cursor_val === field_separator_char) {
if (in_quote === false) {
if (field_start_index <= field_end_index) {
result.push(parse_csv_field(csv_line_str.substring(field_start_index, field_end_index)));
field_start_index = csr_index + 1;
} // Else: just ignore field_end_index has not moved or field has not been sliced for parsing the field
} // Else: just ignore because we are in quote
}
csr_index++;
cursor_val = csv_line_str[csr_index];
field_end_index = csr_index;
field_end_index_moved = true;
}
return result;
}
function parse_csv_field(csv_field_str) {
with_quote = (csv_field_str[0] === '"');
if (with_quote) {
csv_field_str = csv_field_str.substring(1, csv_field_str.length - 1); // remove the start and end quotes
csv_field_str = csv_field_str.split('""').join('"'); // handle double quotes
}
return csv_field_str;
}
// Initial method: check the first newline character only
function get_new_line_char(csv_str) {
if (csv_str.indexOf('\r\n') > -1) {
return '\r\n';
} else {
return '\n'
}
}
Just use .split(','):
var str = "How are you doing today?";
var n = str.split(" ");

In Specman, how can I tell if a reference to a unit has the do-not-generate modifier in front of it?

In Specman, how can I tell if a reference to a unit has the do-not-generate modifier, '!', at the reference's definition?
e.g.
unit foo_u {
};
extend sys {
foo : foo_u is instance;
foo_ptr_generated : foo_u;
keep foo_ptr_generated == foo;
!foo_ptr_notgenerated : foo_u;
connect_pointers() is also {
foo_ptr_notgenerated = foo;
};
};
Without inspecting the code or relying on a naming convention, how can I tell that foo_ptr_generated went through Specman's constraint solver and foo_ptr_notgenerated was procedurally set?
Finally figured this out. This code will determine which references are generated and which aren't in Specman 6 via the reflection interface:
<'
type my_t : [ A,B,C];
unit foo_u {
sub_typiness : my_t;
other_sub_typiness : my_t;
when A'sub_typiness foo_u {
!bar_ptr : bar_u;
};
when B'other_sub_typiness foo_u {
in_b_sub_type : int;
};
when A'sub_typiness B'other_sub_typiness foo_u {
in_a_b_subtype :int;
};
b : bah_u is instance;
};
unit bar_u {
sub_typiness : my_t;
other_sub_typiness :my_t;
when B'sub_typiness bar_u {
!foo_ptr : foo_u;
};
when C'other_sub_typiness bar_u {
inc_sub_type : int;
};
when A'other_sub_typiness bar_u {
!you_shouldnt_see_this_field : bah_u;
};
when B'sub_typiness C'other_sub_typiness bar_u{
in_b_c_subtype :int;
};
!b : bah_u;
b2 : bah_u;
};
unit bah_u {
};
extend any_unit {
!path_to_parent :string;
!my_e_path : string;
post_generate() is also {
if not me is a sys {
path_to_parent = get_parent_unit().e_path();
};
my_e_path = e_path();
};
print_pointers() is {
for each ( wf ) in rf_manager.get_struct_of_instance(me).as_a(rf_like_struct).sd_.all_when_fields {
if not str_match(wf.name,"/___/"){
};
};
for each (sub_unit) in get_all_units(any_unit) {
sub_unit.print_pointers();
};
};
};
struct wank_s {
oh_joy: int;
};
extend sys {
foo : A'sub_typiness B'other_sub_typiness foo_u is instance;
bar : B'sub_typiness C'other_sub_typiness bar_u is instance;
keep bar.b2 == foo.b;
wank : wank_s;
connect_pointers() is also {
foo.bar_ptr = bar;
bar.foo_ptr = foo;
bar.b = foo.b;
};
!possible_ptr : any_unit;
!is_a_unit : bool;
!an_e_path : string;
!a_unit : any_unit;
!a_field : field;
--
-- rf_manager.get_all_unit_instances
my_specman( cmd : string ) : bool is {
try {
--out(cmd);
specman(cmd);
} else {
return FALSE;
};
return TRUE;
};
check_generation() is also {
out("PRINT OUT FIELDS");
var all_units : list of any_unit;
for each any_unit(u) in get_all_units(any_unit) {
for each ( wf ) in rf_manager.get_struct_of_instance(u).as_a(rf_like_struct).sd_.all_when_fields {
--print wf.source_ref;
--print wf.source_ref.to_string().as_a(uint);
--if wf.source_ref.to_string().as_a(uint) > 1000 and not str_match(wf.name,"/___/"){
if not str_match(wf.name,"/___|driver_trans/"){
an_e_path = append(u.e_path(),".",wf.name);
an_e_path = append(an_e_path, " - marked_as_reachable=",wf.fgi.marked_as_reachable);
if wf.base_type is a struct_descriptor (s){
an_e_path = append(an_e_path, " - instantiated == ", s.instantiated);
};
if wf.base_type is a struct_descriptor (s) and not s is a list_descriptor and not s is a long_descriptor {
sys.is_a_unit = FALSE;
if not my_specman(append("sys.is_a_unit = ( ",u.e_path(),".",wf.name,
" != NULL)")) {
sys.is_a_unit = FALSE;
};
if sys.is_a_unit {
if not my_specman(append(" sys.is_a_unit = (",u.e_path(),".",wf.name,
".as_a(any_struct) == ",u.e_path(),".",wf.name,".get_enclosing_unit(any_unit).as_a(any_struct))")) {
sys.is_a_unit = FALSE;
};
};
if sys.is_a_unit and str_match(wf.name,"/([\w_]+'+[\w_']*)[\w_]+/") {
if not my_specman( append( "sys.is_a_unit = ( ",u.e_path(),
" is a ", $1, " ",
rf_manager.get_struct_of_instance(u).as_a(rf_like_struct).sd_.name,
")")) {
sys.is_a_unit = FALSE;
};
};
if is_a_unit {
sys.a_unit = NULL;
if not my_specman(append("sys.a_unit = (",u.e_path(),".",wf.name,
".as_a(any_unit))")) {
sys.a_unit = NULL;
};
if sys.a_unit != NULL {
sys.an_e_path = NULL;
var printout := append(":: ",u.e_path(), ".", wf.name);
compute my_specman(append( "sys.an_e_path = ",u.e_path(),".",wf.name,
".my_e_path"));
if append(u.e_path(),".",wf.name) != sys.an_e_path {
printout = append(printout, " -> ");
printout = append(printout, sys.an_e_path);
if wf.fgi != NULL and wf.fgi.gcs.size() > 0 and (
wf.fgi.gcs[0].last_reduction != UNDEF or
wf.fgi.gcs[0].gcois.size() > 0 ) {
printout = append(printout, " GENERATED POINTER");
};
};
out(printout);
};
};
};
--out( " ",an_e_path);
--specman(append("print ",u.e_path(),".",wf.name));
};
};
--for each ( m ) in rf_manager.get_struct_of_instance(u).as_a(rf_like_struct).sd_.methods {
-- if m != NULL {
-- out(m.md.name);
-- };
--};
--print u.get_all_attribute_names();
--print u.get_attribute("agent");
--print u.agent_kind();
-- for each (m) in rf_manager.get_struct_of_instance(u).get_declared_methods() {
-- print m;
-- };
-- out("fields at '",u.e_path(),"'s struct level:");
-- for each (f) in rf_manager.get_struct_of_instance(u).get_declared_fields() {
-- out( " ",u.e_path()," -> ",f.f_.short_name);
-- };
-- out("fields at '",u.e_path(),"'s exact sub-type level:");
-- for each (f) in rf_manager.get_exact_subtype_of_instance(u).get_declared_fields() {
-- out( " ",u.e_path()," -> ",f.f_.short_name);
-- };
-- out("");
};
-- out("\nPRINT OUT UNIT INSTANCES");
-- for each any_unit(u) in {sys;foo;bar} {
-- for each (i) in rf_manager.get_all_unit_instances(u) {
-- if i != u {
-- out(u.e_path(), " has unit ", i.e_path());
-- };
-- };
-- };
};
};
'>
Running with:
specman -c 'load rf_test.e; gen'
Produces:
Generating the test using seed 1...
PRINT OUT FIELDS
:: sys.bar.B'sub_typiness'foo_ptr -> sys.foo
:: sys.bar.b -> sys.foo.b
:: sys.bar.b2 -> sys.foo.b GENERATED POINTER
:: sys.foo.A'sub_typiness'bar_ptr -> sys.bar
:: sys.foo.b
:: sys.logger.base_unit -> sys
:: sys.logger
:: sys.foo
:: sys.bar
Starting the test ...
Running the test ...