I am interfacing SD card(16Gb Sandisk ultra micro SD) to STM32F407 micro-controller with SDIO protocol using chan FatFS library. When I try to write data into existing file, f_write returns FR_OK and returns numbers of written bytes(this value is equal to number of bytes to write), but f_close() returns FR_DISK_ERR, and in the end the file is empty.
With more experiment i found that if i format the micro SD card using 64Kb unit allocation size and the existing file with some text in it then i am able to write 64Kb data to file but f_close() returns FR_DISK_ERR, and in the end the file is not empty. I am able to see the data in Windows 10 OS.
If the existing file has no text in this then i am getting an empty file even though f_write is returning FR_OK but f_close is returning FR_DISK_ERR.
In short when I try to use f_write on a text file i created from my PC I can overwrite the content of that file till 64Kb. But i can't get it to work with an empty file i created with f_open
I came across a similar post with the same issue
TMS320F2812 FatFs f_write returns FR_DISK_ERR
I tried the solutions given in the above post but it didn't work. Since i have 192K RAM in my controller, i guess its sufficient enough for this FatFs module to work. My stack size is around 13Kb and Heap size 4Kb which is too much for this application. SD card is given 3.3V supply voltage.
I went little deep in code to see where the error is occurring and found that i am getting SD_ILLEGAL_CMD error while setting block size for card. Inside f_close(ff.c file)->f_sync(ff.c file)->move_window(ff.c file)->disk_read(diskio.c file)->SD_ReadBlock(sdcard.c file) is returning SD_ILLEGAL_CMD error while setting block size for card.
Any solutions are appreciated. If more information is required please feel free to ask, i will update with more information.
Chan FatFs Version - R0.07e
Related
Recently, I am testing the proper usage of ext4 filesystem. what is my expert is that:
when system crashed, the data had been write return ok can not loss, but metadate can.
Here is my usage:
1. call fallocate to alloc centain space
fallocate(fd, 0, 0, 4*1024*1024); //4MB
2. call fsync(fd) let data and metadata write to disks
3. then i call function to randomly write the file with 4k size(random data but not 0). with O_DRICT flagļ¼but not call fsync. I log the offset with return write ok.
4. check the offset that logged. but i find in some offset, read 4k data, is 0. It seems mean that offset isn't used like hole files.
My question is that:
<1. why after calling fallocate and fsync the metadata of the file still seems
indicate some blocks is not used, so when read it return null. It is my understand .
<2. have other api to call, can make sure that in allocate space with file is not holes ,after that when write data return ok with O_DIRECT can make sure the data will not be loss even the system crashed.
Thanks.
Only writing to the file space can eliminate the hole. Without writing, there is no dirty page and fsync simply does nothing.
I am wondering how did you execute you step 4. It seems that you did it by a manual crash, did you? If you read it after write without a crash, it should not be zero, provided you wrote non-zeros. If you read it after a crash, zero can happen if disk cache existed. However, this kind of zero is not like holes, they are zeros read from the disk (very probably the disk contains zeros).
I have test code that starts with a large existing file on SD Card, test.dat, and the test does this:
f_open(&fp, "test.dat", FA_OPEN_ALWAYS|FA_READ|FA_WRITE);
f_write(&fp, bufferOfZeros, 4096, &ioBytes);
Seems straight forward.
The result of the f_write is SD Card corruption.
I've traced the EDMA on my embedded device to watch FatFs read/write requests. The f_open read requests are all healthy, they properly find the file and FAT table.
The f_write starts by reading the first sector into scratch buffer, great. Then it memcpy's the zeros into the scratch buffer, great. After memcpy of 512 zeros, it needs to commit the sector and move to the next one.
It's at this point where the code is confused. It writes that scratch buffer of zeros to the FAT table!!
The offending code in ff.c/f_write() is:
if (fp->flag & FA__DIRTY) { /* Write-back sector cache */
if (disk_write(fp->fs->drv, fp->buf, fp->dsect, 1) != RES_OK)
ABORT(fp->fs, FR_DISK_ERR);
fp->flag &= ~FA__DIRTY;
Here, fp->buf is the sector buffer of zeros, but fp->dsect is the FAT sector (8318), not the file data sector (10240)! Oops. At disk_write() fp->buf matches fp->sect.
Seems like a basic use case here, I am shocked to see this.
I'm hoping someone in the world has anyone dug into a problem like this before with FatFs?
Clifford is correct - it's a problem on my end, and I knew such a simple use case has to work.
I switched the test to
f_open(&fp, "test.dat", FA_OPEN_ALWAYS|FA_READ|FA_WRITE);
f_write(&fp, pattern, 512, &ioBytes);
f_lseek(&fp, 0);
f_read(&fp, zeroedBuffer, 512, &ioBytes);
This revealed a caching problem. The f_read operation succeeded, and the DMA completed fine, but zeroedBuffer remained unmodified because L0 was not flushed. Therefore, FatFs was seeing a mix of correct values and stale values. I fixed EDMA caching and now it all works.
Another learning, TI Starterware comes with R0.4b. I started with that, and it returned FR_RW_ERROR that was perplexing. I moved up to R0.11a and got much better error feedback.
Anyone know of unit tests for FatFs?
Recently I'm into creating checksums for files in go. My code is working with small and big files. I tried two methods, the first uses ioutil.ReadFile("filename") and the second is working with os.Open("filename").
Examples:
The first function is working with the io/ioutil and works for small files. When I try to copy a big file my ram gets blastet and for a 1.5GB iso it uses 3GB of ram.
func byteCopy(fileToCopy string) {
file, err := ioutil.ReadFile(fileToCopy) //1.5GB file
omg(err) //error handling function
ioutil.WriteFile("2.iso", file, 0777)
os.Remove("2.iso")
}
Even worse when I want to create a checksum with crypto/sha512 and io/ioutil.
It will never finish and abort because it runs out of memory.
func ioutilHash() {
file, _ := ioutil.ReadFile(iso)
h := sha512.New()
fmt.Printf("%x", h.Sum(file))
}
When using the function below everything works fine.
func ioHash() {
f, err := os.Open(iso) //iso is a big ~ 1.5tb file
omg(err) //error handling function
defer f.Close()
h := sha512.New()
io.Copy(h, f)
fmt.Printf("%x", h.Sum(nil))
}
My Question:
Why is the ioutil.ReadFile() function not working right? The 1.5GB file should not fill my 16GB of ram. I don't know where to look right now.
Could somebody explain the differences between the methods? I don't get it with reading the go-doc and examples.
Having usable code is nice, but understanding why its working is way above that.
Thanks in advance!
The following code doesn't do what you think it does.
func ioutilHash() {
file, _ := ioutil.ReadFile(iso)
h := sha512.New()
fmt.Printf("%x", h.Sum(file))
}
This first reads your 1.5GB iso. As jnml pointed out, it continuously makes bigger and bigger buffers to fill it. In the end, And total buffer size is no less than 1.5GB and no greater than 1.875GB (by the current implementation).
However, after that you then make another buffer! h.Sum(file) doesn't hash file. It appends the current hash to file! This may or may not cause yet another allocation.
The real problem is that you are taking that file, now appended with the hash, and printing it with %x. Fmt actually pre-computes using the same type of method jnml pointed out that ioutil.ReadAll used. So it constantly allocated bigger and bigger buffers to store the hex of your file. Since each letter is 4 bits, that means we are talking about no less than a 3GB buffer for that and no greater than 3.75GB.
This means your active buffers may be as big 5.625GB. Combine that with the GC not being perfect and not removing all the intermediate buffers, and it could very easily fill your space.
The correct way to write that code would have been.
func ioutilHash() {
file, _ := ioutil.ReadFile(iso)
h := sha512.New()
h.Write(file)
fmt.Printf("%x", h.Sum(nil))
}
This doesn't do nearly the number the allocations.
The bottom line is that ReadFile is rarely what you want to use. IO streaming (using readers and writers) is always the best way when it is an option. Not only do you allocate much less when you use io.Copy, you also hash and read the disk concurrently. In your ReadFile example, the two resources are used synchronously when they don't depend on each other.
ioutil.ReadFile is working right. It's your fault to abuse the system resources by using that function for things you know are huge.
ioutil.ReadFile is a handy helper for files you're pretty sure in advance that they're going to be small. Like configuration files, most source code files etc. (Actually it's optimizing things for files <= 1e9 bytes, but that's an implementation detail and not part of the API contract. Your 1.5GB file forces it to use slice growing and thus allocating more than one big buffer for your data in the process of reading the file.)
Even your other approach using os.File is not okay. You definitely should be using the "bufio" package for sequential processing of large files, see bufio.NewReader.
I am doing a firmware upgrade using SPI bus on EEPROM as well as Internal ROM of 8051, basically writing a .hex file on both these memory devices.I am able to see .hex file written there.I am able to see slave and master are communicating properly, but not able to write anything on my memory devices.
If you have suggestions and if you have faced similar problems, please let me know where is the actual problem.
Any inputs would be welcomed.
Regards,
Ravi
I think more information will likely be required. In any case, here a few pitfalls I could see:
Hex Files are not necessarily memory images. The 8051s I've worked with usually use Intel Hex which is an ASCII format that describes the memory. The format is well documented here.
If you're having trouble writing to the EEPROM, you may not be writing the proper instructions. Typically, SPI EEPROM will be Byte addressed, but still has paging internally. You should start your writes on a Page boundary and write the whole page, then issue another write command, etc. By convention if you overrun a page, or start in the middle of a page it will loop around. So if your page is 8 bytes long, and you start writing 0-7 starting at index 4, you'll get:
Page Start: Index 0 = 4
Index 1 = 5
Index 2 = 6
Index 3 = 7
Index 4 = 0
Index 5 = 1
Index 6 = 2
Index 7 = 3
Most EEPROMs have locking mechanisms to prevent accidental writes once they are finalized. If the lock has been set, you will need to write an unlocking method (this will be detailed in the data sheet if it has it)
To further help you, please reference part numbers and better yet Data Sheets if you can.
I'm trying to write a program to log the float values of samples in an audio file. My steps are as follows:
Open an Extended Audio File
Set up audio format (AudioStreamBasicDescription)
Apply audio format to my Extended Audio File
Set up an AudioBufferList
Read Extended Audio File into AudioBufferList with ExtAudioFileRead()
Log float values of AudioBufferList
I've put the entire 90 line class in this gist: https://gist.github.com/792630
The trouble is, if I apply an audio format to my Extended Audio File (Step 3), I get an error trying to read the file (Step 5). If I comment out step 5, the file reads fine but I will not have enforced my format for the read and I don't get floats when logging.
Any suggestions would be greatly appreciated. Thanks!
One thing I noticed right away is that you are allocating an AudioBufferList on the stack but setting mNumberBuffers to 2. It's fine to use ABLs on the stack, but if you do that they can only ever contain one buffer. But since you've set the client format to mono that isn't your real problem.
The real problem is that you aren't passing the address of fileRef to ExtAudioFileOpenURL- you're passing the value- so there is no way that the call can initialize the variable properly.
The call should look like this:
CheckResult(ExtAudioFileOpenURL(inputFileURL, &fileRef), "ExtAudioFileOpenURL failed");
I did that, compiled your code and everything worked fine.