I am working on loading csv file from dataloader in datastax graph.
My csv files structure is following
first file(Year_2015.txt)
YearID
second file(BaseVehicle_2005.txt)
BaseVehicleID|YearID|MakeID|ModelID
for first file I have create vertex level as year and key as YearID for second I have create vertex level as BaseVehicle while key as BaseVehicleID and ignore YearID,MakeID,ModelID. Now I want to create edge between second(BaseVehicle) and first(Year) with edge level year and property YearID but nothing has work for me. Please let me know what i need to change?
The documentation has examples : https://docs.datastax.com/en/latest-dse/datastax_enterprise/graph/dgl/dglCSV.html
Below is a sample loader script that works on JSON data, but it shows how to load both Vertices and Edges from the same record, using mappers to decide which elements to use.
config load_threads: 8
config batch_size: 1000
config load_new: true
config driver_retry_attempts: 10
config driver_retry_delay: 1000
/** SAMPLE INPUT
{"actor_name":"'Bear'Boyd, Steven","title_name":"The Replacements","year":"2000","role":"Defensive Tackle - Washington Sentinels","episode":"The Replacements"}
*/
input = File.json(filename )
//Defines the mapping from input record to graph elements
actorMapper = {
key "actor_name" // the unique id for an actor
label "actor"
ignore "title_name"
ignore "role"
ignore "year"
ignore "episode"
}
titleMapper = {
key "title_name" // the unique id for a title
label "title"
ignore "sex"
ignore "actor_name"
ignore "role"
ignore "episode"
}
castMapper = {
label "cast"
outV "actor_name", {
label "actor"
key "actor_name"
}
inV "title_name", {
label "title"
key "title_name"
}
ignore "year"
ignore "sex"
// remaining should be edge properties
// pickup role as property
// pickup episode as property
}
//Load vertex records from the input according to the mapping
load(input).asVertices(actorMapper)
load(input).asVertices(titleMapper)
load(input).asEdges(castMapper)
Related
source data: Array<File>
target return data: List<Pair<File,File>>
The source data (Array<File>) contains a list of jpegs of a book in the form of (Scan0001.jpg, Scan0002.jpg, ..., Scan000n.jpg). The first file (Scan0001.jpg) is always the front cover of the book and the last file (Scan000n.jpg) is always the back cover of the book. The variable files in the following code snippet is an Array<File!> that only contains jpeg files in the form of Scanxxxx.jpg.
I want to create Pairs of Files of the pages, with the following rules:
1) The covers (front, back) should be always be Pair<File,null> (File being Scan0001.jpg, Scan000n.jpg respectively)
2) If the non-cover pages are uneven (meaning the last page doesn't have a pair) it should be Pair<File,null> (File being Scan000n-1.jpg)
3) The front cover should always be the first File Pair and the back cover always be the last
The following code works, but I feel there is space for improvement in terms of a more efficient or more clean code
val files = selectedFolder.listFiles()
val preliminaryResult = files.toMutableList()
val result = mutableListOf<Pair<File?,File?>>()
result.add(Pair(preliminaryResult.first(),null))
preliminaryResult.removeAt(0)
result.add(Pair(preliminaryResult.last(),null))
preliminaryResult.removeAt(preliminaryResult.size-1)
result.addAll(preliminaryResult.map{
if(preliminaryResult.indexOf(it) % 2 == 0) {
Pair(it,preliminaryResult.getOrNull(preliminaryResult.indexOf(it)+1))
} else {
Pair(null,null)
}
})
result.removeAll{
it == Pair(null,null)
}
result.add(result[1])
result.removeAt(1)
You can insert the nulls you need first so you can use zipWithNext uninterrupted.
val result = selectedFolder.listFiles().toMutableList<File?>().apply {
add(1, null) // for front cover
if (0 == size % 2)
add(size - 1, null) // for odd inner last page
add(null) // for back cover
}.zipWithNext()
.run { slice(indices step 2) }
After reading this excellent Medium article, I've been stoked on Migrations in CraftCMS. They've been super useful in importing the same changes across the 10 or so developers who work on our site.
When trying to change the handles on individual fields within block types within matrix blocks (whew) via migrations, I came across a hurdle. The field itself can easily have its "handle" attribute updated, but the columns table for that matrix block's content (matrixcontent_xxxxx) do not get updated to reflect any updated handles. The association between the Matrix Block & its associated Matrix Content table only exists in the info column in the field record for that Matrix Block.
If the Matrix Block's field is updated via the CMS, the change is reflected, so it's gotta be somewhere in Craft's source, but it's not apparent through the Craft CMS Class Reference.
Any ideas?
Edited to add the migration snippet:
public function safeUp()
{
// Strings for labels, etc.
$matrix_block_instructions = "instructions";
$block_label = "Video";
$block_handle = "video";
$field_handle = "video_url";
$field_label = "Video URL";
$field_instructions = "Add a YouTube or Facebook video URL.";
// Fetching the MatrixBlock fields used on the entries themselves
$video_gallery_1 = Craft::$app->fields->getFieldByHandle("handle_1");
$video_gallery_2 = Craft::$app->fields->getFieldByHandle("handle_2");
$video_gallery_3 = Craft::$app->fields->getFieldByHandle("handle_3");
$galleries = [$video_gallery_1, $video_gallery_2, $video_gallery_3];
foreach( $galleries as $gallery ) {
// Fetching the record for this specific MatrixBlock field.
$gallery_record = \craft\records\Field::find()->where(
['id' => $gallery->id]
)->one();
// Fetching the record for this specific MatrixBlockType
$gallery_block_id = $gallery->getBlockTypes()[0]->id;
$gallery_block = \craft\records\MatrixBlockType::find()->where(
['id' => $gallery_block_id]
)->one();
// Assigning standard labels for the MatrixBlockType
$gallery_block->name = $block_label;
$gallery_block->handle = $block_handle;
$gallery_block->update();
// Getting the specific ... 1 ... field to edit
$field_group = \craft\records\FieldLayout::find()->where(
['id' => $gallery_block->fieldLayoutId]
)->one();
$field_layout_field = $field_group->fields[0];
$field = $field_layout_field->field;
$field = \craft\records\Field::find()->where(
['id' => $field->id]
)->one();
// Assigning standard labels for the Label
$field->name = $field_label;
$field->handle = $field_handle;
$field->instructions = $field_instructions;
$field->update();
// Updating the MatrixBlock record with new instructions
$gallery_record->refresh();
$gallery_record->instructions = $matrix_block_instructions;
$gallery_record->update();
}
OK, so my apologies if anyone was stoked on figuring this out, but my approach above was kind of a crazy person's approach, but I've found my own solution.
The key here is that I should have been interacting with craft\fields\MatrixBlock and the craft\fields\PlainText objects, not craft\records\Field objects. There's a method within \craft\services\Fields for saving the field that requires a FieldInterface implemented. This is actually the default classes returned, and I was making more work for myself in the code!
Within that foreach loop, this worked out:
// Fetching the record for this specific MatrixBlock field.
$gallery->instructions = $matrix_block_instructions;
// Update the MatrixBlockType
$gallery_block_id = $gallery->getBlockTypes()[0]->id;
$gallery_block = \craft\records\MatrixBlockType::find()->where(
['id' => $gallery_block_id]
)->one();
$gallery_block->name = $block_label;
$gallery_block->handle = $block_handle;
$gallery_block->update();
// Update the actual field.
$field = $gallery->blockTypeFields[0];
$field->name = $field_label;
$field->handle = $field_handle;
$field->instructions = $field_instructions;
Craft::$app->getFields()->saveField( $field );
Craft::$app->getFields()->saveField( $gallery );
Thanks for looking at this, and sorry for being a crazy person.
I managed to create an own inline content element (on tt_content table) but when i try to get the values on the frontend via fluid, i get nothing.
I debugged the {data} variable and on the column that my data are saved, there is an integer. I suppose it reads the number of the content elements which were created on the foreign table (accordion). How can i get those values?
At this point the {data} variables reads the tt_content table and the column that has the integer reads the number of content elements on the table accordion.
I suppose no code is necessary. If it is necessary, feel free to comment the part of the code you would like to review.
Best regards
You need to add a DataProcessor to your TypoScript creating the content element, which fetch your accordion records. Example:
tt_content {
yourContentElementName < lib.contentElement
yourContentElementName.templateName = YourContentElementName
yourContentElementName.dataProcessing {
10 = TYPO3\CMS\Frontend\DataProcessing\DatabaseQueryProcessor
10 {
if.isTrue.field = fieldInTtContentWithInteger
table = your_accordion_table
pidInList = this
where.field = uid
where.intval = 1
where.dataWrap = field_pointing_to_ttcontent_record = |
as = accordions
}
}
}
Recently I read this guide on undocumented featuers in TensorFlow, as I needed to pass variable length sequences as input. However, I found the protocol for tf.train.SequenceExample relatively confusing (especially due to lack of documentation), and managed to build an input pipe using tf.train.Example just fine instead.
Are there any advantages to using tf.train.SequenceExample? Using the standard example protocol when there is a dedicated one for variable length sequences seems like a cheat, but does it bear any consequence?
Here are the definitions of the Example and SequenceExample protocol buffers, and all the protos they may contain:
message BytesList { repeated bytes value = 1; }
message FloatList { repeated float value = 1 [packed = true]; }
message Int64List { repeated int64 value = 1 [packed = true]; }
message Feature {
oneof kind {
BytesList bytes_list = 1;
FloatList float_list = 2;
Int64List int64_list = 3;
}
};
message Features { map<string, Feature> feature = 1; };
message Example { Features features = 1; };
message FeatureList { repeated Feature feature = 1; };
message FeatureLists { map<string, FeatureList> feature_list = 1; };
message SequenceExample {
Features context = 1;
FeatureLists feature_lists = 2;
};
An Example contains a Features, which contains a mapping from feature name to Feature, which contains either a bytes list, or a float list or an int64 list.
A SequenceExample also contains a Features, but it also contains a FeatureLists, which contains a mapping from list name to FeatureList, which contains a list of Feature. So it can do everything an Example can do, and more. But do you really need that extra functionality? What does it do?
Since each Feature contains a list of values, a FeatureList is a list of lists. And that's the key: if you need lists of lists of values, then you need SequenceExample.
For example, if you handle text, you can represent it as one big string:
from tensorflow.train import BytesList
BytesList(value=[b"This is the first sentence. And here's another."])
Or you could represent it as a list of words and tokens:
BytesList(value=[b"This", b"is", b"the", b"first", b"sentence", b".", b"And", b"here",
b"'s", b"another", b"."])
Or you could represent each sentence separately. That's where you would need a list of lists:
from tensorflow.train import BytesList, Feature, FeatureList
s1 = BytesList(value=[b"This", b"is", b"the", b"first", b"sentence", b"."])
s2 = BytesList(value=[b"And", b"here", b"'s", b"another", b"."])
fl = FeatureList(feature=[Feature(bytes_list=s1), Feature(bytes_list=s2)])
Then create the SequenceExample:
from tensorflow.train import SequenceExample, FeatureLists
seq = SequenceExample(feature_lists=FeatureLists(feature_list={
"sentences": fl
}))
And you can serialize it and perhaps save it to a TFRecord file.
data = seq.SerializeToString()
Later, when you read the data, you can parse it using tf.io.parse_single_sequence_example().
The link you provided lists some benefits. You can see how parse_single_sequence_example is used here https://github.com/tensorflow/magenta/blob/master/magenta/common/sequence_example_lib.py
If you managed to get the data into your model with Example, it should be fine. SequenceExample just gives a little more structure to your data and some utilities for working with it.
I have this response in soapUI:
<pointsCriteria>
<calculatorLabel>Have you registered for inContact, signed up for marketing news from FNB/RMB Private Bank, updated your contact details and chosen to receive your statements</calculatorLabel>
<description>Be registered for inContact, allow us to communicate with you (i.e. update your marketing consent to 'Yes'), receive your statements via email and keep your contact information up to date</description>
<grades>
<points>0</points>
<value>No</value>
</grades>
<grades>
<points>1000</points>
<value>Yes</value>
</grades>
<label>Marketing consent given and Online Contact details updated in last 12 months</label>
<name>c21_mrktng_cnsnt_cntct_cmb_point</name>
</pointsCriteria>
There are many many many pointsCriteria and I use the below xquery to give me the DB value and Range of what that field is meant to be:
<return>
{
for $x in //pointsCriteria
return <DBRange>
<db>{data($x/name/text())}</db>
<points>{data($x//points/text())}</points>
</DBRange>
}
</return>
And i get the below response
<return><DBRange><db>c21_mrktng_cnsnt_cntct_cmb_point</db><points>0 1000</points></DBRange>
That last bit sits in a property transfer. I need SQL to bring back all rows where that DB field is not in that points range (field can only be 0 or 1000 in this case), my problem is I dont know how to loop through each DBRange/DBrange in this manner? please help
I'm not sure that I really understand your question, however I think that you want to make queries in your DB using specific table with a column name defined in your <db> field of your xml, and using as values the values defined in <points> field of the same xml.
So you can try using a groovy TestStep, first parse your Xml and get back your column name, and your points. To iterate over points if the values are separated with a blank space you can make a split(" ") to get a list and then use each() to iterate over the points on this list. Then using groovy.sql.Sql you can perform the queries in your DB.
Only one more thing, you need to put the JDBC drivers for your vendor DB in $SOAPUI_HOME/bin/ext and then restart SOAPUI in order that it can load the necessary driver classes.
So the follow code approach can achieve your goal:
import groovy.sql.Sql
import groovy.util.XmlSlurper
// soapui groovy testStep requires that first register your
// db vendor drivers, as example I use oracle drivers...
com.eviware.soapui.support.GroovyUtils.registerJdbcDriver( "oracle.jdbc.driver.OracleDriver")
// connection properties db (example for oracle data base)
def db = [
url : 'jdbc:oracle:thin:#db_host:d_bport/db_name',
username : 'yourUser',
password : '********',
driver : 'oracle.jdbc.driver.OracleDriver'
]
// create the db instance
def sql = Sql.newInstance("${db.url}", "${db.username}", "${db.password}","${db.driver}")
def result = '''<return>
<DBRange>
<db>c21_mrktng_cnsnt_cntct_cmb_point</db>
<points>0 1000</points>
</DBRange>
</return>'''
def resXml = new XmlSlurper().parseText(result)
// get the field
def field = resXml.DBRange.db.text()
// get the points
def points = resXml.DBRange.points.text()
// points are separated by blank space,
// so split to get an array with the points
def pointList = points.split(" ")
// for each point make your query
pointList.each {
def sqlResult = sql.rows "select * from your_table where ${field} = ?",[it]
log.info sqlResult
}
sql.close();
Hope this helps,
Thanks again for your help #albciff, I had to add this into a multidimensional array (I renamed field to column and result is a large return from the Xquery above)
def resXml = new XmlSlurper().parseText(result)
//get the columns and points ranges
def Column = resXml.DBRange.db*.text()
def Points = resXml.DBRange.points*.text()
//sorting it all out into a multidimensional array (index per index)
count = 0
bigList = Column.collect
{
[it, Points[count++]]
}
//iterating through the array
bigList.each
{//creating two smaller lists and making it readable for sql part later
def column = it[0]
def points = it[1]
//further splitting the points to test each
pointList = points.split(" ")
pointList.each
{//test each points range per column
def sqlResult = sql.rows "select * from my_table where ${column} <> ",[it]
log.info sqlResult
}
}
sql.close();
return;