If I have a setup with Azure blob storage (usiung AzureReader2 plugin), and I also use the DiskCache plugin, where is the cached image actually stored?
For example, I have an image as http://test.blob.core.windows.net/images/test.jpg, so the resized version is http://test.azurewebsites.net/azure/images/test.jpg?width=50. If I am also using the DiskCache plugin, is a cached version of this request created? And if so where, in the Azure space? If this is the case, I am concerned that this space (usually reserved for the Azure web site) will be geneally quite limited.
Unless the cached version is created in the blob storage itself, in which case everything is perfect
I used FTP to access the Azure website, and there stands imagecache folder in all its glory.
Related
I building a new course-like web application. There will be plenty of images, video and sound files.
I am wondering about possible strategies for static file management between app environments.
My current approach is to use SQL database to store image urls, which will be uploaded via admin panel on the website. The images are to be stored in a blob-like storage (AWS S3 bucket).
This however, when doing changes, requires to upload the image to each environment or create a data migration dev -> staging -> prod in a deployment pipeline.
Am I missing something here? Even if I store files in a single place (single storage account) for all environments, I still need to migrate the database records when making changes to the course.
Should I just apply the changes in prod and create some basic migration data for dev/uat course testing?.
To emphasize, files will only be uploaded by an admin, not by a user. For example, admin uploads the image via admin panel and the image will be automatically included in the course.
I am not sure what's the appropriate way of doing this to manage and test changes properly. If I allow to do this on prod directly without migration, then I'm running the risk of uploading something invalid into the course with untested changes. On the other hand , I am not sure if it's common to migrate SQL data between databases and it will also have it's own pitfalls.
I have used blob storage as a file storage account in .NET Core Web application hosted on Azure app service(PaaS).
My requirement is to create .zip files and then attach to email where it requires UNC path for attachment.
Here I have one option to use app service local storage for temporary file creation and use in attachment.
I am searching other option to map blob storage to any virtual drive in cloud and get its UNC path or any other option?
Also, Can you please suggest what are the possible options to map Azure Blob storage drive in network? I know the following one - App service local storage, VM, Local machine network drive.
First of all, you need to know the concept of UNC path, and then azure webapp can be regarded as a virtual machine in essence, and azure blob storage can also be regarded as a machine. Therefore, it is not feasible to send mail directly through azure blob.
Suggestion:
1. I check the information, you can try azure files to store files and use them.
I think this should be the fastest way, without using other azure products.
Download the file to the project directory, you can create a temporary folder, such as: MailTempFolder, you can download the file from the blob to this folder, and then you can get the UNC path to send mail.
After the sending is successful, just delete the file, it will not occupy too much space of the azure webapp, even if the sending fails, you can still get the zip file without downloading it again.
According to Microsoft's documentation it is possible to upload a python wheel file so that you can use custom libraries in Synapse Analytics.
Here is that documentation: https://learn.microsoft.com/en-us/azure/synapse-analytics/spark/apache-spark-azure-portal-add-libraries
I have created a simple library with just a hello world function that I was able to install with pip on my own computer. So I know my wheel file works.
I uploaded my wheel file to the location Microsoft's documentation say to upload the file.
I also found a youtube video of a person doing exactly what I am trying to do.
Here is the video: https://www.youtube.com/watch?v=t4-2i1sPD4U
Microsoft's documentation mentions this, "Custom packages can be added or modified between sessions. However, you will need to wait for the pool and session to restart to see the updated package."
As far as I can tell there is no way to restart a pool, and I also do not know how to tell if the pool is down or has restarted.
When I try to use the library in a notebook I get a module not found error.
Scaling up or down will force the cluster to restart .
Making changes to the spark pool's scale settings does restart the spark pool as HimanshuSinha-msft suggested. That was not my problem though.
The actual problem was that I needed the Storage Blob Data Contributor role in the data lake storage the files were stored in. I assumed because I already had owner permissions and because I could create a folder and upload there I had all the permissions I needed. Once I got the Storage Blob Data Contributor role though everything worked.
We have an existing setup of AEM 6.1 which uses TarMK for data storage. To migrate the all assets to S3, I followed all steps here: https://docs.adobe.com/docs/en/aem/6-1/deploy/platform/data-store-config.html#Data%20Store%20Configurations (Amazon S3 Data Store). Apparently, the data synced to S3 but when I checked the disk usage report, I still see that assets are using disk space even for existing and newly added assets. What's the purpose of using S3 for assets if they still use the disk space? Or am I doing something wrong? How can I verify that my setup is really using S3? Here is my S3DataStore.config
accessKey="xxxxxxxxxx"
secretKey="xxxxxxxxxx"
s3Bucket="dev-aem-assets-local"
s3Region="eu-west-1"
connectionTimeout="120000"
socketTimeout="120000"
maxConnections="40"
writeThreads="30"
maxErrorRetry="10"
continueOnAsyncUploadFailure=B"true"
cacheSize="0"
minRecordLength="10"
Another question is: Do I need to do the same setup on publisher? Or is it ok just to do it on author and use publisher as is by replicating the binary data?
There are a few parts to your questiob so I'll break down the answer into logical blocks. Shout if I miss anything.
Your setup for migration is correct and S3 will use disk space. This is for the write-through cache.
AEM uses write-through cache for writing to S3 and all the settings for this cache are in your S3 config file. Any writes to data store are first written to this cache. Asynchronous background threads then uploaded to the S3 bucket. This mechanism makes AEM very responsive as it's not blocked by slow S3 writes. Also, data reads for recently written blobs are fast because they don't need slow reads from S3. In short, S3 IO traffic is too slow for AEM so this cache boosts the performance. You cannot disable it as it is required for asynchronous write to S3. You can reduce the size but it's recommended to be at least 50% of your S3 bucket size.
You can verify your S3 setup by looking at your logs for messages related to AWS (grep for aws).
As for publisher, yes you need to migrate from your old publisher to the new publisher. Assuming that you are not using binary-less replication, you will need a different S3 bucket for your publisher. In general, you migrate from author to author and publisher to publisher for a standard implementation.
You can also verify your S3 dat usage by looking at the S3 bucket and the traffic on it. If versioning is enabled on your S3 bucket all the blobs will show version stamping.
Async upload of blobs can be monitored from logs and IP traffic monitoring will show activities related to your S3 bucket. The most useful way is to see the network traffic between your AEM server and S3 end-point.
I am moving my MVC4 application static data like(CSS,IMG,Javascript,etc) to Azure Blob storage enabled with CDN.
It will offload my server and in return will increase system Performance and responsiveness.
I have two queries here :
Will migrating static data to Blob actually increase Performance as compared to MVC4 bundling feature to deliver the content to browser.
What & How to accommodate the code changes involved in the process.How to change the Virtual Path to the Actual corresponding blob URI.
Thanks in advance !
1. Will migrating static data to Blob actually increase Performance as compared to MVC4 bundling feature to deliver the content to browser.
CDN and bundling will both improve performance. CDN will offload the load on your server and serve up the file from the closest data center thus ensuring high availability etc etc. With bundling the first time the page is loaded you will see it loading faster than if the files were not bundled. However once the page as been requested the browser caches the assets. Look at this question as well
What & How to accommodate the code changes involved in the process.How to change the Virtual Path to the Actual corresponding blob URI.
You will have to use the CDN url with the container name and blob name. Remember to have URL version suffixes that you can take off a config file or change if you use the CDN since any changes to your assets will not be reflected in the CDN unless you explicitly force it to refresh with a new version.