I found in the android, data partition and system partition of ext4 file system block group, only 0,1,3,5,7 etc have block group descriptor and super block. But in the Linux is all some, can you tell me how to make ext4 file system block group have block group descriptor and super block
this fact is due to the "sparse_super feature flag" set in the ext4 filesystem on your android device.
sparse_super feature means that the superblock and group-descriptor table will not be part of every block group. Instead it will just be copied to the block group 0 and 1 and all block groups whose number is a power of 3,5,7
-> So this means it will show up in: 0,1,3,5,7,9,25,27,49...
You can dissable this feature on a filesystem with:
tune2fs -s 0 /dev/...
Or while making a new ext4 filesystem:
mke2fs -T ext4 -O ^sparse_super /dev/...
Related
Purpose:
Save a program that writes data to disk from vain attempts of writing to a full filesystem;
Save bandwidth (don't download if nowhere to store);
Save user's and programmer's time and nerves (notify them of the problem instead of having them tearing out their hair with reading misleading error messages and "why the heck this software is not working!").
The question comes in 2 parts:
Reporting storage space statistics (available, used, total etc.), either of all filesystems or of the filesystem that path in question belongs to.
Reporting a filesystem error on running out of space.
Part 1
Share please NATIVE Raku alternative(s) (TIMTOWTDIBSCINABTE "Tim Toady Bicarbonate") to:
raku -e 'qqx{ df -P $*CWD }.print'
Here, raku -executes df (disk free) external program via shell quoting with interpolation qqx{}, feeding -Portable-format argument and $*CWD Current Working Directory, then .prints the df's output.
The snippet initially had been written as raku -e 'qqx{ df -hP $*CWD }.print' — with both -human-readable and -Portable — but it turned out that it is not a ubiquitously valid command. In OpenBSD 7.0, it exits with an error: df: -h and -i are incompatible with -P.
For adding human-readability, you may consider Number::Bytes::Human module
raku -e 'run <<df -hP $*CWD>>'
If you're just outputting what df gives you on STDOUT, you don't need to do anything.
The << >> are double quoting words, so that the $*CWD will be interpolated.
Part 1 — Reporting storage space statistics
There's no built in function for reporting storage space statistics. Options include:
Write Raku code (a few lines) that uses NativeCall to invoke a platform / filesystem specific system call (such as statvfs()) and uses the information returned by that call.
Use a suitable Raku library. FileSystem::Capacity picks and runs an external program for you, and then makes its resulting data available in a portable form.
Use run (or similar1) to invoke a specific external program such as df.
Use an Inline::* foreign language adaptor to enable invoking of a foreign PL's solution for reporting storage space statistics, and use the info it provides.2
Part 2 — Reporting running out of space
Raku seems to neatly report No space left on device:
> spurt '/tmp/failwrite', 'filesystem is full!'
Failed to write bytes to filehandle: No space left on device
in block <unit> at <unknown file> line 1
> mkdir '/tmp/failmkdir'
Failed to create directory '/tmp/failmkdir' with mode '0o777': Failed to mkdir: No space left on device
in block <unit> at <unknown file> line 1
(Programmers will need to avoid throwing away these exceptions.)
Footnotes
1 run runs an external command without involving a shell. This guarantees that the risks attendant with involving a shell are eliminated. That said, Raku also supports use of a shell (because that can be convenient and appropriate in some scenarios). See the exchange of comments under the question (eg this one) for some brief discussion of this, and the shell doc for a summary of the risk:
All shell metacharacters are interpreted by the shell, including pipes, redirects, environment variable substitutions and so on. Shell escapes are a severe security concern and can cause confusion with unusual file names. Use run if you want to be safe.
2 Foreign language adaptors for Raku (Raku modules in the Inline:: namespace) allow Raku code to use code written in other languages. These adaptors are not part of the Raku language standard, and most are barely experimental status, if that, but, conversely, the best are in great shape and allow Raku code to use foreign libraries as if they were written for Raku. (As of 2021 Inline::Perl5 is the most polished.)
I have 3 groups of photos, from 3 different cameras (with time sychronised onboard all cameras) but with different naming schemes (e.g.: IMG_3142.jpg, DCM_022.jpg). I would like to rename every photo file with the following naming convention:
1_yyyy_mm_dd_hh_mm_ss.jpg for earliest
2_yyyy_mm_dd_hh_mm_ss.jpg for next earliest, and so on,
until we reach around 5000_yyyy_mm_dd_hh_mm_ss.jpg for the last one (i.e. the most recent)
I would like the yyyy_mm_dd_hh_mm_ss field to be replaced by the “date and time taken” value for when this photo was taken. Which is saved in the metadata/properties of each file.
I have seen awk used to carry out similar operations but I'm not familiar enough to know how to access the “time taken” metadata, etc.
Also, not that this should make a difference: my computer is a mac.
You can use jhead for this. The command is:
jhead -n%Y_%m_%d_%H_%M_%S *.jpg
Make a COPY of your files first before running it! You can install jhead with homebrew using:
brew install jhead
Or, if you don't have homebrew, download here for OS X.
That will get you the date in the filename as you wish. The sequence number is a little more difficult. Try what I am suggesting above and, if you are happy with that, we can work on the sequence number maybe. Basically, you would run jhead again to set the file modification times of your files to match the time they were shot - then the files can be made to show up in the listing in date order and we can put your sequence number on the front.
So, to get the file's date set on the computer to match the time it was taken, do:
jhead -ft *.jpg
Now all the files will be dated on your computer to match the time the photos were taken. Then we need to whizz through them in a loop with our script adding in the sequence number:
#!/bin/bash
seq=1
# List files in order, oldest first
for f in $(ls -rt *jpg)
do
# Work out new name
new="$seq_$f"
echo Rename $f as $new
# Remove "#" from start of following command if things look good so the renaming is actually done
# mv "$f" $new"
((seq++))
done
You would save that in your HOME directory as renamer, then you would go into Terminal and make the script executable like this:
chmod +x renamer
Then you need to go to where your photos are, say Desktop\Photos
cd "$HOME/Desktop/Photos"
and run the script
$HOME/renamer
That should do it.
By the way, I wonder how wise it is to use a simple sequence number at the start of your filenames because that will not make them come up in order when you look at them in Finder.
Think of file 20 i.e. 20_2015_02_03_11:45:52.jpg. Now imagine that files starting with 100-199 will be listed BEFORE file 2o, also files 1000-1999 will also be listed before file 20 - because their leading 1s come before file 20's leading 2. So, you may want to name your files:
0001_...
0002_...
0003_...
...
0019_...
0020_...
then they will come up in sequential order in Finder. If you want that, use this script instead:
#!/bin/bash
seq=1
for f in $(ls -rt *jpg)
do
# Generate new name with zero-padded sequence number
new=$(printf "%04d_$f" $seq)
echo Rename $f as $new
# Remove "#" from start of following command if things look good so the renaming is actually done
# mv "$f" $new"
((seq++))
done
I am writing a utility that walks a directory tree on Mac OS X (10.6 and higher) and tries to detect changes that have occurred since the directory was last synchronized with a back-up location.
When I initially synchronize the files and folders I obtain the inode number and store it in the database record for that file or folder:
NSString *oldFilePath = /* ... */;
NSError *error = nil;
NSDictionary *attributes = [[NSFileManager defaultManager] attributesOfItemAtPath:oldFilePath error:&error];
/* set database record for oldFilePath to [attributes fileSystemFileNumber] */
When I encounter a new file or folder I first do a database lookup using the inode number to find the original file, if any.
But in the case where a file has moved from a parent directory to a sub-directory, and I am trying to detect changes to the parent directory I would like to be able to use the saved inode number to identify the new path so that I can distinguish between a move and a delete.
On Mac the GetFileInfo command performs a reverse lookup of inode numbers.
GetFileInfo /.vol/234881029/344711
Should produce:
file: "/path/to/file"
...
Martin R's answer only works on directories.
inode numbers are only unique within a filesystem, so you need at least device and inode number to identify a file.
On the HFS+ file system, the inode number is in fact identical to the "Macintosh File Id", and there is a special "/.vol" filesystem that allows you to find a directory by device and inode.
Example:
$ cd /.vol/234881029/342711
$ pwd
/Volumes/Data/tmpwork/test20/test20.xcodeproj
$ stat .
234881029 342711 drwxr-xr-x 5 martin staff 0 170 ......
As you can see, 234881029 is the device number of "/Volumes/Data", 342711 is the inode number of "tmpwork/test20/test20.xcodeproj" within that filesystem, and
cd /.vol/<deviceNo>/<inodeNo>
transferred you directly to that folder. You could now use getcwd() to determine the real path to that folder.
The "/.vol" filesystem is documented in the legacy Technical Q&A QA1113.
Disclaimer: I tried this only on OS X 10.7 and I am fairly sure that it works on older systems. I have no idea if you can rely on this feature in future versions of OS X. Also it is very HFS specific.
On Unix-like systems, many filenames may reference the same inode, so you'd have to search the filesystem. I don't know if MacOS provides a shortcut.
Note that, as explained above, the /.vol/ 'magic' directory needs the device ID for the volume, and the inode of the directory or file. You can get the device ID of the volume as the first number returned by stat as explained in a different answer here.
# stat returns device ID as '234881026' and confirms inode is '32659974'
~$ stat /Volumes/Foo
234881026 32659974 lrwxr-xr-x 1 root admin 0 1 ... /Volumes/Foo
# access file using ./vol/<device ID>/<inode>
~$ cd /.vol/234881026/1017800
:../Prague 2011 March$
~$ GetFileInfo /.vol/234881026/1017800/IMG_3731.JPG
file: "/Users/roger/Pictures/Prague 2011 March/IMG_3731.JPG"
I'm using two batch files, and I need to control variables in one of them, from the other. Is this possible?
You cannot directly influence one process' environment from another process. You know, we've kinda outgrown ye olde days of real mode by now :-)
This all depends a bit on what you're trying to achieve here. If you're calling one batch file from the other, as in
call second.cmd
then the called one »inherits« the environment of the parent batch. So any variable you defined earlier will continue to exist in the child batch. You cannot propagate changes up to the parent, though and you cannot change a variable in the child batch after it has been started, too. It might still be a viable option if all you need is to perform some one-time initialization before starting the child batch.
What you could do is to agree on a file used by both batch files that they will use as a means of communicating with each other, likely located in the temporary directory. Each batch file would need to regularly check for the file to be present and if so, read it and update its variables accordingly. For that to succeed you need points in the batch files where they can look for that file. The simplest would be two files that simply do a bit communication with each other:
The code for that is here:
callchat.cmd:
#echo off
set SENDFILE=%TEMP%\1.out
set RCVFILE=%TEMP%\1.in
start call chat.cmd
ping -w 5000 -n 1 123.45.67.89 >nul 2>&1
set RCVFILE=%TEMP%\1.out
set SENDFILE=%TEMP%\1.in
start call chat.cmd
chat.cmd:
#echo off
setlocal enabledelayedexpansion
rem Prevent direct use
if not defined SENDFILE goto :eof
if not defined RCVFILE goto :eof
set MESSAGE0=I don't know what to say ...
set MESSAGE1=Foo
set MESSAGE2=Bar
set MESSAGE3=Hey there!
set MESSAGE4=Meow.
:loop
rem wait a bit
ping -n 1 -w 1000 123.45.67.89 >nul 2>&1
rem look whether we need to show something
if exist %RCVFILE% (
for /f "delims=" %%l in (%RCVFILE%) do echo Received message at %TIME% - %%l
del "%RCVFILE%"
)
rem randomly send out messages. Roughly ever three times we try this
set /a msg=%random% %% 5
set msg=!MESSAGE%msg%!
if %RANDOM% LSS 10000 (
>>%SENDFILE% echo(%msg%
echo(Sent message "%msg%"
)
goto loop
The batch file is started twice, with different input/output files – in fact, the reversed role of the files from the first invocation. Then it's little more than an endless loop that looks into its input file and read what's in there and writing stuff to its output file (which is the input file for the other batch).
I had to introduce a delay in starting both of them to avoid the PRNGs for both being exactly the same. It also reduced the cases where access to the file failed (this could probably be alleviated by renaming it before reading from it – or, if writing longer content, renaming it to its final name only after being done writing). It's just a simple demo application to show you that it might be possible that way.
To set environment variables you wouldn't print out what's in the file but call it as a batch file, for example:
if exist %RCVFILE% call %RCVFILE%
It would need the proper extension for that, though. You can also read it line by line and have each line contain a VARIABLE=VALUE pair:
if exist %RCVFILE% call for /f "tokens=1* delims==" %%a in (%RCVFILE%) do set %%a=%%b
The techniques mentioned above for improving reliability when accessing the same file from two different programs still apply.
As mentioned, this is only a rough idea how you could operate, iff I understood your question correctly.
I'm using rsync to make a backup of my server files, and I have two questions:
In the middle of the process I need to stop and start rsync again.
Will rsync start from the point where it stopped or it will restart from the beginning?
In the log files I see "f+++++++++". What does it mean?
e.g.:
2010/12/21 08:28:37 [4537] >f.st...... iddd/logs/website-production-access_log
2010/12/21 08:29:11 [4537] >f.st...... iddd/web/website/production/shared/log/production.log
2010/12/21 08:29:14 [4537] .d..t...... iddd/web/website/production/shared/sessions/
2010/12/21 08:29:14 [4537] >f+++++++++ iddd/web/website/production/shared/sessions/ruby_sess.017a771cc19b18cd
2010/12/21 08:29:14 [4537] >f+++++++++ iddd/web/website/production/shared/sessions/ruby_sess.01eade9d317ca79a
Let's take a look at how rsync works and better understand the cryptic result lines:
1 - A huge advantage of rsync is that after an interruption the next time it continues smoothly.
The next rsync invocation will not transfer the files again, that it had already transferred, if they were not changed in the meantime. But it will start checking all the files again from the beginning to find out, as it is not aware that it had been interrupted.
2 - Each character is a code that can be translated if you read the section for -i, --itemize-changes in man rsync
Decoding your example log file from the question:
>f.st......
> - the item is received
f - it is a regular file
s - the file size is different
t - the time stamp is different
.d..t......
. - the item is not being updated (though it might have attributes
that are being modified)
d - it is a directory
t - the time stamp is different
>f+++++++++
> - the item is received
f - a regular file
+++++++++ - this is a newly created item
The relevant part of the rsync man page:
-i, --itemize-changes
Requests a simple itemized list of the changes that are being made to
each file, including attribute changes. This is exactly the same as
specifying --out-format='%i %n%L'. If you repeat the option, unchanged
files will also be output, but only if the receiving rsync is at least
version 2.6.7 (you can use -vv with older versions of rsync, but that
also turns on the output of other verbose messages).
The "%i" escape has a cryptic output that is 11 letters long. The
general format is like the string YXcstpoguax, where Y is replaced by
the type of update being done, X is replaced by the file-type, and the
other letters represent attributes that may be output if they are
being modified.
The update types that replace the Y are as follows:
A < means that a file is being transferred to the remote host (sent).
A > means that a file is being transferred to the local host (received).
A c means that a local change/creation is occurring for the item (such as the creation of a directory or the changing of a symlink,
etc.).
A h means that the item is a hard link to another item (requires --hard-links).
A . means that the item is not being updated (though it might have attributes that are being modified).
A * means that the rest of the itemized-output area contains a message (e.g. "deleting").
The file-types that replace the X are: f for a file, a d for a
directory, an L for a symlink, a D for a device, and a S for a
special file (e.g. named sockets and fifos).
The other letters in the string above are the actual letters that will
be output if the associated attribute for the item is being updated or
a "." for no change. Three exceptions to this are: (1) a newly created
item replaces each letter with a "+", (2) an identical item replaces
the dots with spaces, and (3) an unknown attribute replaces each
letter with a "?" (this can happen when talking to an older rsync).
The attribute that is associated with each letter is as follows:
A c means either that a regular file has a different checksum (requires --checksum) or that a symlink, device, or special file has a
changed value. Note that if you are sending files to an rsync prior to
3.0.1, this change flag will be present only for checksum-differing regular files.
A s means the size of a regular file is different and will be updated by the file transfer.
A t means the modification time is different and is being updated to the sender’s value (requires --times). An alternate value of T
means that the modification time will be set to the transfer time,
which happens when a file/symlink/device is updated without --times
and when a symlink is changed and the receiver can’t set its time.
(Note: when using an rsync 3.0.0 client, you might see the s flag
combined with t instead of the proper T flag for this time-setting
failure.)
A p means the permissions are different and are being updated to the sender’s value (requires --perms).
An o means the owner is different and is being updated to the sender’s value (requires --owner and super-user privileges).
A g means the group is different and is being updated to the sender’s value (requires --group and the authority to set the group).
The u slot is reserved for future use.
The a means that the ACL information changed.
The x means that the extended attribute information changed.
One other output is possible: when deleting files, the "%i" will
output the string "*deleting" for each item that is being removed
(assuming that you are talking to a recent enough rsync that it logs
deletions instead of outputting them as a verbose message).
Some time back, I needed to understand the rsync output for a script that I was writing. During the process of writing that script I googled around and came to what #mit had written above. I used that information, as well as documentation from other sources, to create my own primer on the bit flags and how to get rsync to output bit flags for all actions (it does not do this by default).
I am posting that information here in hopes that it helps others who (like me) stumble up on this page via search and need a better explanation of rsync.
With the combination of the --itemize-changes flag and the -vvv flag, rsync gives us detailed output of all file system changes that were identified in the source directory when compared to the target directory. The bit flags produced by rsync can then be decoded to determine what changed. To decode each bit's meaning, use the following table.
Explanation of each bit position and value in rsync's output:
YXcstpoguax path/to/file
|||||||||||
||||||||||╰- x: The extended attribute information changed
|||||||||╰-- a: The ACL information changed
||||||||╰--- u: The u slot is reserved for future use
|||||||╰---- g: Group is different
||||||╰----- o: Owner is different
|||||╰------ p: Permission are different
||||╰------- t: Modification time is different
|||╰-------- s: Size is different
||╰--------- c: Different checksum (for regular files), or
|| changed value (for symlinks, devices, and special files)
|╰---------- the file type:
| f: for a file,
| d: for a directory,
| L: for a symlink,
| D: for a device,
| S: for a special file (e.g. named sockets and fifos)
╰----------- the type of update being done::
<: file is being transferred to the remote host (sent)
>: file is being transferred to the local host (received)
c: local change/creation for the item, such as:
- the creation of a directory
- the changing of a symlink,
- etc.
h: the item is a hard link to another item (requires
--hard-links).
.: the item is not being updated (though it might have
attributes that are being modified)
*: means that the rest of the itemized-output area contains
a message (e.g. "deleting")
Some example output from rsync for various scenarios:
>f+++++++++ some/dir/new-file.txt
.f....og..x some/dir/existing-file-with-changed-owner-and-group.txt
.f........x some/dir/existing-file-with-changed-unnamed-attribute.txt
>f...p....x some/dir/existing-file-with-changed-permissions.txt
>f..t..g..x some/dir/existing-file-with-changed-time-and-group.txt
>f.s......x some/dir/existing-file-with-changed-size.txt
>f.st.....x some/dir/existing-file-with-changed-size-and-time-stamp.txt
cd+++++++++ some/dir/new-directory/
.d....og... some/dir/existing-directory-with-changed-owner-and-group/
.d..t...... some/dir/existing-directory-with-different-time-stamp/
Capturing rsync's output (focused on the bit flags):
In my experimentation, both the --itemize-changes flag and the -vvv flag are needed to get rsync to output an entry for all file system changes. Without the triple verbose (-vvv) flag, I was not seeing directory, link and device changes listed. It is worth experimenting with your version of rsync to make sure that it is observing and noting all that you expected.
One handy use of this technique is to add the --dry-run flag to the command and collect the change list, as determined by rsync, into a variable (without making any changes) so you can do some processing on the list yourself. Something like the following would capture the output in a variable:
file_system_changes=$(rsync --archive --acls --xattrs \
--checksum --dry-run \
--itemize-changes -vvv \
"/some/source-path/" \
"/some/destination-path/" \
| grep -E '^(\.|>|<|c|h|\*).......... .')
In the example above, the (stdout) output from rsync is redirected to grep (via stdin) so we can isolate only the lines that contain bit flags.
Processing the captured output:
The contents of the variable can then be logged for later use or immediately iterated over for items of interest. I use this exact tactic in the script I wrote during researching more about rsync. You can look at the script (https://github.com/jmmitchell/movestough) for examples of post-processing the captured output to isolate new files, duplicate files (same name, same contents), file collisions (same name, different contents), as well as the changes in subdirectory structures.
1.) It will "restart the sync", but it will not transfer files that are the same size and timestamp etc. It first builds up a list of files to transfer and during this stage it will see that it has already transferred some files and will skip them. You should tell rsync to preserve the timestamps etc. (e.g. using rsync -a ...)
While rsync is transferring a file, it will call it something like .filename.XYZABC instead of filename. Then when it has finished transferring that file it will rename it. So, if you kill rsync while it is transferring a large file, you will have to use the --partial option to continue the transfer instead of starting from scratch.
2.) I don't know what that is. Can you paste some examples?
EDIT: As per http://ubuntuforums.org/showthread.php?t=1342171 those codes are defined in the rsync man page in section for the the -i, --itemize-changes option.
Fixed part if my answer based on Joao's