I'm currently struggling while implementing a Spider for a Web Application, because my pages are not well structured at all. There're fields on the page, but sometimes they don't appear, and it's difficult to distinguish those because they only use class, not id as identifier. Is there a method to optimize the way we get the data in pages?
Below, an example of one page to crawl :
<div class = 'view-activity-field-wrapper even' style = 'display:none' >
<div class="view-activity-label">Status Notes <span><img src="/images/helpIcon.png" alt="" width="8" height="10" align="absmiddle" data-tooltip="stickyStatusNotes" /></span>
</div>
<div class="view-activity-field"></div>
</div>
<div style = 'clear:both'></div>
<div class = 'view-activity-field-wrapper odd' style = 'display:none' >
<div class="view-activity-label">Relevant Question <span><img src="/images/helpIcon.png" alt="" width="8" height="10" align="absmiddle" data-tooltip="stickyRelevantQuestion" /></span>
</div>
<div class="view-activity-field"></div>
</div>
<div style = 'clear:both'></div>
<!-- KEEP VALUE PROVIDED HERE -->
<div class = 'view-activity-field-wrapper odd' style = 'display:none' >
<div class="view-activity-label">Value Provided <span><img src="/images/helpIcon.png" alt="" width="8" height="10" align="absmiddle" data-tooltip="viewvalueprovided" /></span>
</div>
<div class="view-activity-field"></div>
</div>
<div style = 'clear:both'></div>
Related
I am trying to output the value of individual checkboxes and also add a class to the label when the checkbox is checked. I can do one of the other but not both together. If I add :value="attendance" the output works as individual instances but the adding of the class doesn't work and if I add value="attendance" then it treats the 2 checkboxes as one value.
Can someone help please?
<div class="container">
<div class="row">
<div class="col-sm">
<label
class="btn btn-outline-primary label-in-filter"
:class="{
showattendances:
showattendancesisChecked('attendance'),
}"
v-for="attendance in uniqueattendances"
:key="attendance"
>
<!-- <input
value="attendance"
id="attendance"
name="showattendances"
class="ck-in-filter"
type="checkbox"
v-model="attendances"
/> -->
<input
id="attendance"
name="showattendances"
class="ck-in-filter"
type="checkbox"
:value="attendance"
v-model="attendances"
/>
{{ attendance }}
</label>
</div>
<p v-for="attendance in attendances" :key="attendance">
{{ attendance }}
</p>
</div>
</div>
methods: {
showattendancesisChecked(value) {
return this.attendances.includes(value);
},}
I have an Ajax form on my view. It relates to Newsletter
<div class="d-flex flex-column">
<div class="d-flex align-items-center">
<div class="px-lg-8">
<div class="d-flex align-items-center">
<div class=" ml-3">
<svg xmlns="http://www.w3.org/2000/svg" width="21.238" height="15.291" viewBox="0 0 21.238 15.291">
<path d="M0 0v15.292h21.238V0zm.849.849h19.54v1.062l-8.31 7.244-.04.04a2.194 2.194 0 0 1-1.42.571 2.2 2.2 0 0 1-1.42-.571c-.158-.138-1.293-1.118-2.111-1.832C4.661 5.255.974 2.021.849 1.912zm0 2.19c.737.642 3.35 2.917 5.575 4.858L.849 12.305zm19.539 0v9.266l-5.575-4.407c2.226-1.943 4.839-4.216 5.576-4.858zM7.075 8.469l1.566 1.367.013.013a3.057 3.057 0 0 0 1.965.77 3.051 3.051 0 0 0 1.978-.783c.12-.1 1.059-.913 1.58-1.367l6.212 4.911v1.062H.849V13.38z" data-name="Path 319"></path>
</svg>
</div>
<h5 class="mb-0"> #_localizer["NewsLetterMemberShip"]</h5>
</div>
<p class="my-3 ">#_localizer["ReceiveNewsLetter"]</p>
<div class="textbox-footer ">
<form asp-controller="Home" asp-action="NewsLetter" data-ajax="true" data-ajax-mode="replace" data-ajax-success="NewLetterPostSuccess">
<input asp-for="Email" id="NewsLetterEmail" class="form-control input-textbox-footer "
placeholder="#_localizer["EnterEmail"]">
<span asp-validation-for="Email"></span>
<button type="submit" class="btn btn--orange footer__send-btn">#_localizer["SendButton"]</button>
</form>
</div>
</div>
</div>
</div>
It works well. but now I see after click on newsletter ,It inserts email twice,
It has a reference:
<script src="/lib/jquery-unobtrusive-ajax/jquery.unobtrusive-ajax.min.js"></script>
If I delete this refrence, it will be work correct and insert 1 time.
This is my script :
function NewLetterPostSuccess(result) {
swal('info', result.text, 'info');
$('#NewsLetterEmail').val('');
}
I see the way for resolve is delete this reference.
But is not for ajax form?
and If I delete reference I have another misktake too.
How can resolve this problem?
I find my mistake .
It referenced to page twice with "unobtrusive" at 2 different Route.
<script src="/lib/jquery-unobtrusive-ajax/jquery.unobtrusive-ajax.min.js"></script>
I deleted one of them and It works well now.
Check all your project for the reference always! :)
I am using scrapy shell just to make sure my selectors for my spider are correct. I am able to get all other sections I need except this one p tag that contains the cross ref part numbers. I am scraping from this particular page here
When I try response.css('div.col-1-2-2' > div.rpr-help m-chm > div > p::text').extract() it returns blank
When I try response.css('div > p::text').extract() the results have the section I am looking for plus a bunch of data I do not want.
I have a feeling this is going to be a super easy answer, but I have no idea what I am missing here
This is a snippet of the html section of the page I am trying to scrape, the last 'p' tag starting with Part Number
<div class="col-1-2-2">
<div id="img-detail" style="text-align:center;">
<div id="img-detail-main">
<a id="ctl00_cphMain_imgenlarge" rel="nofollow" href="/detail-img.aspx?id=3094537&i=" class="cboxElement"><img id="ctl00_cphMain_iMain" src="https://cdn.appliancepartspros.com/images/product/cache/whirlpool-clutch-assembly-285785-ap3094537_01_l.jpg" style="border-width:0px;outline:none;">
<div class="img-overlay" style="display:none;"><img src="/images/play.png" style="height:107px;"></div>
<div id="main-text-overlay" style="display:none;"></div>
</a>
</div>
<div class="img-help">Click image to open expanded view</div>
<div id="img-detail-thumb">
<div class="a-button a-active">
<img id="ctl00_cphMain_rImgTh_ctl01_imgTh" src="https://cdn.appliancepartspros.com/images/product/cache/whirlpool-clutch-assembly-285785-ap3094537_01_tt.jpg" style="border-width:0px;">
</div>
<div class="a-button">
<img id="ctl00_cphMain_rImgTh_ctl02_imgTh" src="https://cdn.appliancepartspros.com/images/product/cache/whirlpool-clutch-assembly-285785-ap3094537_02_tt.jpg" style="border-width:0px;">
</div>
<div class="a-button">
<img id="ctl00_cphMain_rImgTh_ctl03_imgTh" src="https://cdn.appliancepartspros.com/images/product/cache/whirlpool-clutch-assembly-285785-ap3094537_03_tt.jpg" style="border-width:0px;">
</div>
<div class="a-button">
<img id="ctl00_cphMain_rImgTh_ctl04_imgTh" src="https://cdn.appliancepartspros.com/images/product/cache/whirlpool-clutch-assembly-285785-ap3094537_04_tt.jpg" style="border-width:0px;">
</div>
<div class="a-button">
<img id="ctl00_cphMain_rImgTh_ctl05_imgTh" src="https://cdn.appliancepartspros.com/images/product/cache/whirlpool-clutch-assembly-285785-ap3094537_05_tt.jpg" style="border-width:0px;">
</div>
<div class="a-button">
<img id="ctl00_cphMain_rImgTh_ctl06_imgTh" class="diagram" data-dcmt="Clutch assembly AP3094537 is number 5 on this diagram. This is to give you an idea of the appearance and the location of the part. Your appliance model may be slightly different." src="https://483cda5f439700fab03b-6195bc77e724f6265ff507b1dc015ddb.ssl.cf1.rackcdn.com/0029384112_4.gif" style="border-width:0px;">
</div>
<div class="a-button">
<img id="ctl00_cphMain_rImgTh_ctl07_imgTh" class="video" src="https://img.youtube.com/vi/7RS1l6t8efc/hqdefault.jpg" style="border-width:0px;">
<div class="img-overlay"><img src="/images/play.png"></div>
</div>
</div>
</div>
<div class="rpr-help m-chm">
<div class="header">
<h2 class="h6">Repair Help</h2>
</div><!-- /end .header -->
<div class="inner m-bsc">
<ul>
<li>Repair Video</li>
<li>Repair Q&A</li>
</ul>
</div>
<div>
<br>
<span class="h4">Cross Reference Information</span><br>
<p>Part Number 285785 (AP3094537) replaces 2670, 285331, 285380, 285422, 285540, 285761, 285785VP, 3350015, 3350114, 3350115, 3351342, 3351343, 387888, 388948, 388949, 3946794, 3946847, 3951311, 3951312, 62699, 63174, 63765, 64176, AH334641, EA334641, J27-662, LP326, PS334641.
<br>
</p>
</div>
</div>
</div>
Hope this works
response.xpath('//div[#class="col-1-2-2"]//p/text()').extract_first()
You can try this also, response.xpath('(//div[#class="rpr-help m-chm"]//p//text())[1]').get()
I'm trying to focus on several elements of my form but the first one, despite being applied, returns an error by console.
This is my template:
<div class="container">
<div class="col-xs-12">
<div class="row">
<h1 class="animal-title">Your selection is : </h1>
</div>
<div class="wrapper">
<form class="first-form" #submit.prevent="onSubmit">
<div class="image-wrapper">
<div class="sel-image">
<div v-on:click="imageSelected = true" v-for="item in items" v-bind:key="item.id">
<label>
<input
type="radio"
name="selectedItem"
ref="item"
:value="item.id"
v-model="itemFormInfo.selectedItem"
#change="onChangeItem($event)"
/>
<img v-if="item.id === 1" src="../../assets/1.png" />
<img v-if="item.id === 2" src="../../assets/2.png" />
<img v-if="item.id === 3" src="../../assets/3.png" />
</label>
<p class="cie-animal-subtitle">{{item.name}}</p>
</div>
</div>
</div>
<div class="form-select">
<div v-show="filteredStock && (imageSelected || itemFormInfo.selectedItem) > 0">
<h1 v-if="this.itemName === 'Phone' || this.itemName === 'Tablet'" for="selectedItem" ref="itemVisible">
Select the brand of your <span>{{this.itemName}}</span> :
</h1>
<h1 v-if="this.itemName === 'PC'" for="selectedBreed" ref="itemVisible">
Select the type of your <span>{{this.itemName}}</span> :
</h1>
<select
ref="brand"
class="form-control"
id="selectedBrand"
v-model="itemFormInfo.selectedBrand"
#change="onChangeBrand($event)">
<option v-for="brand in filteredBrand" v-bind:key="brand.name">{{ brand.name }}</option>
</select>
<div v-show="this.isBrandSelected">
<h1>What are you going to use your
<span>{{itemName}}</span> for ?
</h1>
<input
type="text"
id="componentName"
ref="componentName"
class="form-control fields"
style="text-transform: capitalize"
v-model="itemFormInfo.component"
#keypress="formChange($event)"
/>
<div class="loader-spinner" v-if="loading">
<app-loader/>
</div>
</div>
</div>
</div>
<div class="service-options" v-show="isComponentCompleted">
<div class="from-group">
<h1>
Here are the options for your <span>{{this.itemFormInfo.component}}</span> :
</h1>
<div class="services">
<div class="column-service" v-for="option in options" v-bind:key="option.name">
<div class="service-name">{{option.name}}</div>
<div class="service-price">{{option.price.toString().replace(".", ",")}} </div>
</div>
</div>
and here my first method
onChangeItem(event) {
let item = event.target._value;
this.itemName = this.getItemName(item);
if (this.isItemSelected = true) {
this.isItemSelected = false;
this.isComponentCompleted = false;
this.isLoaderFinished = false;
this.itemFormInfo.name = ""
}
this.$refs.item.focus();
},
in this function that I control my first input, the focus is working but it returns me by console the following error:
"this.$refs.item.focus is not a function at VueComponent.onChangeItem"
I have seen some references to similar cases where they involved the reference in a setTimeout or used the this.$nextTick(() => method but it didn't work in my case.
What am I doing wrong?
How can I focus on the next select with ref brand, once I have chosen the value of the first input?
Thank you all for your time and help in advance
How can I focus on the next select with ref brand, once I have chosen the value of the first input?
You want to put focus on brand but your onChangeItem handler is calling this.$refs.item.focus() (trying to focus item). Seems strange to me...
Reason for the error is you are using ref inside v-for.
Docs: When used on elements/components with v-for, the registered reference will be an Array containing DOM nodes or component instances
So the correct way for accessing item ref will be this.$refs.item[index].focus().
Just be aware that right now v-for refs do not guarantee the same order as your source Array - you can find some workarounds in the issue discussion...
I am working on an ASP.NET MVC 4 Project. I want to style data validation errors on my login page with Bootstrap 3.0. When I debug the page and it gives data validation errors, this codes are disappeared in source of my login form:
<form action="/Account/Login" class="col-md-4 col-md-offset-4 form-horizontal well" method="post"><input name="__RequestVerificationToken" type="hidden" value="Zbg4kEVwyQf87IWj_L4alhiHBIpoWRCJ9mRWXF6syGH4ehg9idjJCqRrQTMGjONnywMGJhMFmGCQWWvBbMdmGFSUPqXpx6XaS4YfpnbFm8U1" /><div class="validation-summary-errors"><ul><li>The user name or password provided is incorrect.</li>
</ul></div> <div class="form-group control-group">
<div class="col-md-6 col-md-offset-3">
<input class="input-validation-error form-control" data-val="true" data-val-required="User name alanı gereklidir." id="UserName" name="UserName" placeholder="Kullanıcı Adı" type="text" value="" />
<span class="field-validation-error" data-valmsg-for="UserName" data-valmsg-replace="true">User name alanı gereklidir.</span>
</div>
</div>
<div class="form-group">
<div class="col-md-6 col-md-offset-3">
<input class="input-validation-error form-control" data-val="true" data-val-required="Password alanı gereklidir." id="Password" name="Password" placeholder="Şifre" type="password" />
<span class="field-validation-error" data-valmsg-for="Password" data-valmsg-replace="true">Password alanı gereklidir.</span>
</div>
</div>
<div class="form-group">
<div class="col-md-4 col-md-offset-4">
<button class="btn btn-default" type="submit">Giriş Yap</button>
</div>
</div>
</form>
How can I style these errors like "for=inputError" property of label with Bootstrap 3?
As it's shown in Bootstrap's docs, you need to apply class has-error to the div that contains the input and has class form-group:
<div class="form-group has-error">
...
</div>
It's a quite ugly to write a condition for each property you want to check and apply class has-error depending on the results of that condition, though you can do it like so:
<div class="form-group #(Html.ViewData.ModelState.IsValidField(Html.IdFor(x => x.UserName)) ? null : "has-error" )">
This takes care of the server side validation. However, there is also client side validation you need to think about. For that you'd need to write some jQuery that would check for existence of class field-validation-error and apply class has-error depending on the result.
You may do it all your self, though I suggest checking out TwitterBootstrapMVC which does all of that automatically for you. All you'd have to write is:
#Html.Bootstrap().FormGroup().TextBoxFor(m => m.UserName)
Disclaimer: I'm the author of TwitterBootstrapMVC. Using it in Bootstrap 2 is free. For Bootstrap 3 it requires a paid license.