I have several sites, all of them have the same robots.txt.
When I modify one I have to modify others. That so troubles some.
I have an idea that generated robots.txt by back-end code. The back-end will get the data from a remote server by WEB API and generate it.
Here is the code from a tutorial in StackOverflow:
public class OthersController: Controller
{
[Route("/robots.txt")]
public ContentResult RotbotsTXT()
{
String Result=///some code get robots data from remote server.
return this.Content(Result, "text/plain", Encoding.UTF8);
}
}
In spite, it works well on the browser.
However, I met a strange situation.
Some spiders access the route correctly but can not detect it(for example Baidu spider).
And also, some spiders(for example Google bot) will strangely access route www.abc.com//robots.txt (the robots.txt never store in here) but not www.abc.com/robots.txt.
After return to the old way by creating a TXT file, all problems are clear.
What's wrong with my code? How can I solve it? Thank you.
Change [Route("/robots.txt")] to [Route("robots.txt")] like this:
public class OthersController: Controller
{
[Route("robots.txt")]
public ContentResult RobotsTXT()
{
String Result=///some code get robots data from remote server.
return this.Content(Result, "text/plain", Encoding.UTF8);
}
}
Related
I've run into the following problem when porting an app from REST API to GDAA.
The app needs to download some of (thousands of) JPEG images based on user selection. The way this is solved in the app is by downloading a thumbnail version first, using this construct of the REST API:
private static InputStream getCont(String rsid, boolean bBig){
InputStream is = null;
if (rsid != null) try {
File gFl = bBig ?
mGOOSvc.files().get(rsid).setFields("downloadUrl" ).execute():
mGOOSvc.files().get(rsid).setFields("thumbnailLink").execute();
if (gFl != null){
GenericUrl url = new GenericUrl(bBig ? gFl.getDownloadUrl() : gFl.getThumbnailLink());
is = mGOOSvc.getRequestFactory().buildGetRequest(url).execute().getContent();
}
} catch (UserRecoverableAuthIOException uraEx) {
authorize(uraEx.getIntent());
} catch (GoogleAuthIOException gauEx) {}
catch (Exception e) { }
return is;
}
It allows to get either a 'thumbnail' or 'full-blown' version of an image based on the bBig flag. User can select a thumbnail from a list and the full-blown image download follows (all of this supported by disk-base LRU cache, of course).
The problem is, that GDAA does not have an option to ask for reduced size / thumbnail version of an object (AFAIK), so I have to resort to combining both APIs, which makes the code more convoluted then I like (bottom of the page). Needles to state that the 'Resource ID' needed by the REST may not be immediately available.
So, the question is: Is there a way to ask GDAA for a 'thumbnail' version of a document?
Downloading thumbnails isn't currently available in the Drive Android API, and unfortunately I can't give a timeframe to when it will be available. Until that time, the Drive Java Client Library is the best way to get thumbnails on Android.
We'd appreciate if you go ahead and file a feature request against our issue tracker: https://code.google.com/a/google.com/p/apps-api-issues/
That gives requests more visibility to our teams internally, and issues will be marked resolved when we release updates.
Update: I had an error in the discussion of the request fields.
As Ofir says, you can't get thumbnails with the Drive Android API and you can get thumbnails with the Drive Java Client Library. This page has is a really good primer for getting started:
https://developers.google.com/drive/v3/web/quickstart/android
Oddly, I can't get the fields portion of the request to work as it is on that quick start. As I've experienced, you have to request the fields a little differently.
Since you're doing a custom field request you have to be sure to add the other fields you want as well. Here is how I've gotten it to work:
Drive.Files.List request = mService.files()
.list()
.setFields("files/thumbnailLink, files/name, files/mimeType, files/id")
.setQ("Your file param and/or mime query");
FileList files = request.execute();
files.getFiles(); //Each File in the collection will have a valid thumbnailLink
A sample query might be:
"mimeType = 'image/jpeg' or mimeType = 'video/mp4'"
Hope this helps!
I've created a HttpHandler (.ashx) for clients download content (videos) from my website. First I was using the WriteFile method, that I realized it was requiring to much memory and then I decided to change it to TransmitFile method.
But one weird thing happened, I wasn't able to make more than one download anymore. I had to wait a download finishes and start the other.
Basically the code is like this:
System.IO.FileInfo file = new System.IO.FileInfo(file_path);
context.Response.Clear();
if (flagH264)
{
context.Response.ContentType = "video/mp4";
}
else
{
context.Response.ContentType = "video/x-ms-wmv";
}
context.Response.AddHeader("Content-Length", file.Length.ToString());
context.Response.AddHeader("Content-Disposition", "attachment; filename=" + name);
//context.Response.WriteFile(file_path.Trim());
context.Response.TransmitFile(file_path.Trim());
context.Response.Flush();
Anyone may know what is this problem?
I found what was the problem.
The HttpHandler (ashx) I was using for the download page was implementing an interface IRequireSessionState that gave me read/write rights to manipulate session data. When using TransmitFile method IIS blocks any operation on the system to protect session data from being altered.
The solution was changing the IRequireSessionState for IReadOnlySessionState, that gives only reading access to session data and there was no need to provide any kind of security, blocking user actions.
I’d like to restrict access to a folder of controllers that are used for admin purposes only. I’ve tried a number of ways and not coming up with a solution. These controllers are behind password protection. But, I’d like to just remove it from view if someone happens to stumble upon the right directory. Can this be done? I’d rather not do it from htaccess. I have access to the apache config files, so I’d like to handle it there.
Does it have anything to do with the way Codeigniter routes? Or, am I just way off?
This what I’m using that doesn’t work
<Directory /var/www/application/controllers/folder/>
Order deny,allow
Deny from all
Allow from xxx.xxx.xxx.xxx
</Directory>
Due to the way we re-write urls to work with CI, you'd never match your Apache config because you're actually requesting index.php?{args}. If you want to filter, you have to do it in CI instead. Your options are a core controller or hooks.
A simple way to do it is to create a core controller that your admin/ area scripts extend, and check the IP there.
Something like this:
application/core/MY_Controller.php:
class MY_Controller extends CI_Controller
{
public function __construct()
{
parent::__construct();
$this->load->config('permitted_ips');
// check visitor IP against $config['ips'] array, redirect as needed
}
}
Then, in your 'sensitive' controllers, extend MY_Controller:
application/controllers/admin/seekrit.php
class Seekrit extends MY_Controller
{
public function __construct() {
parent::__construct();
/* at this point any invalid IP has been redirected */
}
}
Now, if you're already using a core controller for something else, just check $this->uri->segment() to see if they're in a restricted area before loading the allowed IP configuration file and checking / redirecting / dying or whatever else you need to do.
Also, there's no need to use a constructor in your admin controllers if you don't need one, as the parent will be constructed if one is not defined. Just be sure to call the parent if you define one.
You could also put the white list in a database, Redis, whatever.
Another way to do this would be by using hooks, specifically the pre_controller hook. By the time that hook is entered, all of the security and base classes have run. This would be appropriate if you wanted to protect some or all of your routes in a more granular fashion. There, you could define an array containing routes, such as:
$protected_routes = array(
'foo' => array(
'allow_ip' => '1.2.3.4',
'redirect_if_not' => site_url('goaway')
)
)
Then, in your hook class (or function) match the first segment (my example is just a function):
$CI = get_instance();
$CI->load-config('my_hook');
$protected_routes = $CI->config->item('protected_routes');
$segment = $CI->uri->segment(1); // foo
if (in_array($segment, $protected_routes)) {
// grab $protected_routes[$segment] and work with it
}
This has the advantage of not cluttering up your core controller as many people use that as a means of sharing code between methods. However, the hook will run on every request which means adding another two file loads to bootstrap.
I used the hook method on a large RESTful service to protect certain endpoints by requiring additional headers, and enforcing different kinds of rate limiting on others. Note, the code above is just an example of what could go in the hook, not how to set up the hook itself. Read the hooks section of the CI manual, it's extremely easy and straight forward.
Finally, if you really want to do it via .htaccess, you'll have to go by the request itself. The directory application/controllers/foo is never entered, the actual request is /foo/controller/method{args}, which causes CI to instantiate the foo/controller.php class. Remember, once re-written, the server sees index.php?....
To accomplish this, you can re-write based on the request URI pattern, something like this (have not tested, YMMV):
RewriteRule (^|/)foo(/|$) - [F,L]
Which can be used to redirect anyone accessing the virtual path to your protected controllers. This could be preferable as it prevents PHP from needing to handle it, but you lose the granularity of control over what happens when there is a match. Still, you could use something like the above re-write combined with a hook or core implementation if you have more than one sensitive area to protect.
Tim Post's idea above is similar to another method I either saw on this site or somewhere else. It took me awhile to get back to this issue, but at long last it's done.
As TheShiftExchange pointed out in the comments under my original question, .htaccess will not work for a Codeigniter project. Below is what I ultimately ended up with and seems to work well. It probably isn't 100% secure, but I really just wanted to remove these pages from being directly accessed. If someone were to manage to get to the page there is still a user/pass login screen.
New Config File in application/config
switch (ENVIRONMENT) //set in index.php
{
case 'development':
$config['admin_ips'] = array('XXX.XXX.XXX.XXX');
break;
case 'testing':
$config['admin_ips'] = array('XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX');
break;
case 'production':
$config['admin_ips'] = array('XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX', 'XXX.XXX.XXX.XXX');
break;
}
New Controller
class Admin_IP_Controller extends MY_Controller {
function __construct()
{
parent::__construct();
$this->load->config('admin_ips');
if (!in_array($this->input->ip_address(), $this->config->item('admin_ips')))
{
show_404();
}
}
}
I'm developing a portlet deployed under weblogic server 10. Amongst other functionalities, my portlet need to get some parameters passed in the URL and do something according to each one. Well, my problem is that i can't figure out the right way to get the query string. I have found different approaches on the internet, but none of them seems to work on weblogic. I am able to get the server name, context path or whatever, but no query string..
Does anybody know any solution to this ? Or at least give me a clue ? If i take them from JSP, am i able to pass them over to the .java class ?
Best regards,
Adrian Zaharia
Portlet technology provides two major types of urls - ActionURL and RenderURL.
ActionURL triggers processAction (action phase method) on the target portlet whereas
RenderURL forwards the request to doView (render phase method) on the target portlet.
Also, note that its best to avoid any portlet state change in the render phase.
Typically the jsp passing parameters over to portlet would do...
PortletURL url = renderResponse.createActionURL();
url.setParameter("paramName", "paramVal");
Click Me
The portlet retrieving parameter in processAction or Action Phase would do...
public void processAction(ActionRequest request, ActionResponse response)
throws PortletException, PortletSecurityException, IOException {
....
String paramVal = (String)request.getParameter("paramName");
....
}
I found a better solution for my needs. I was able to get the httprequest like this:
HttpServletRequest httpRequest = (HttpServletRequest) request.getAttribute("javax.servlet.request");
Then get the entire url from a header called referer:
String referer = httpRequest.getHeader("referer");
Thought i should share.
Thanks !
I work at a college and have been developing an ASP.NET site with many, many reports about students, attendance stats... The basis for the data is an MSSQL server DB which is the back end to our student management system. This has a regular maintenance period on Thursday mornings for an unknown length of time (dependent on what has to be done).
Most of the staff are aware of this but the less regular users seem to be forever ringing me up. What is the easiest way to disable the site during maintenance obviously I can just try a DB query to test if it is up but am unsure of the best way to for instance redirect all users to a "The website is down for maintenance" message, bearing in mind they could have started a session prior to the website going down.
Hopefully, something can be implemented globally rather than per page.
Drop an html file called "app_offline.htm" into the root of your virtual directory. Simple as that.
Scott Guthrie on the subject and friendly errors.
I would suggest doing it in Application_PreRequestHandlerExecute instead of after an error occurs. Generally, it'd be best not to enter normal processing if you know your database isn't available. I typically use something like below
void Application_PreRequestHandlerExecute(Object sender, EventArgs e)
{
string sPage = Request.ServerVariables["SCRIPT_NAME"];
if (!sPage.EndsWith("Maintenance.aspx", StringComparison.OrdinalIgnoreCase))
{
//test the database connection
//if it fails then redirect the user to Maintenance.aspx
string connStr = ConfigurationManager.ConnectionString["ConnectionString"].ConnectionString;
SqlConnection conn = new SqlConnection(connStr);
try
{
conn.Open();
}
catch(Exception ex)
{
Session["DBException"] = ex;
Response.Redirect("Maintenance.aspx");
}
finally
{
conn.Close();
}
}
}
You could display a message to people who have logged in saying "the site will be down for maintenance in xxx minutes" then run a service to log everyone out after xxx minutes. Then set a flag somewhere that every page can access, and at the top of every page(or just the template page) you test if that flag is set, if it is, send a redirect header to a site is down for maintenance page.
What happens now when the site is down and someone tries to hit it? Does ADO.NET throw a specific exception you could catch and then redirect to the "website down" page?
You could add a "Global.asax" file to the project, and in its code-behind add an "Application_Error" event handler. It would fire whenever an exception is thrown and goes uncaught, from anywhere in your web app. For example, in C#:
protected void Application_Error(object sender, EventArgs e)
{
Exception e = Server.GetLastError().GetBaseException();
if(e is SqlException)
{
Server.ClearError();
Server.Transfer("~/offline.aspx");
}
}
You could also check the Number property on the exception, though I'm not sure which number(s) would indicate it was unable to connect to the database server. You could test this while it's down, find the SQL error number and look it up online to see if it's specifically what you really want to be checking for.
EDIT: I see what you're saying, petebob.
The "offline.html" page won't work if the user was already navigating within the site, or if he's accessing the site from a bookmark/external link to a specific page.
The solution I use is to create a second web site with the same address (IP or host header(s)), but have it disabled by default. When the website is down, a script deactivates the "real" web site and enables the "maintenance" website instead. When it comes back online, another script switches back to the "real" web site.
The "maintenance" web site is located in a different root directory, with a single page with the message (and any required images/css files)
To have the same message shown on any page, the "maintenance" web site is set up with a 404 error handler that will redirect any request to the same "website is down for maintenance" page.
A slightly more elegant version of the DB check on every page would be to do the check in the Global.asax file or to create a master page that all the other pages inherit from.
The suggestion of having an online site and an offline site is really good, but only really applicable if you have a limited number of sites to manage on the server.
EDIT: Damn, the other answers with these suggestions came up after I loaded the page. I need to remember to refresh before replying :)
James code forgets to close the connection, should probably be:
try
{
conn.Open();
}
catch(Exception ex)
{
Session["DBException"] = ex;
Response.Redirect("Maintenance.aspx");
}
finally
{
conn.Close();
}
Thanks for the replies so far I should point out I'm not the one that does the maintenance nor does I have access all the time to IIS. Also, I prefer options where I do nothing as like all programmers I am a bit lazy.
I know one way is to check a flag on every page but I'm hoping to avoid it. Could I not do something with the global.asax page, in fact, I think posting has engaged my brain:
Think I could put in Application_BeginRequest a bit of code to check the SQL state then redirect:
HttpContext context = HttpContext.Current;
if (!isOnline())
{
context.Response.ClearContent();
context.Response.Write("<script language='javascript'>" +
"top.location='" + Request.ApplicationPath + "/public/Offline.aspx';</scr" + "ipt>");
}
Or something like that may not be perfect not tested yet as I'm not at work. Comments appreciated.