Plugin: CSVGrab

I have used the plugin a number of times in the past without a problem. Now even the most simple use – a single test field in a test weblog – triggers the plugin to grab just the home page of the site and break it up into separate entries.

I have done numerous tests. I’m now starting to wonder if it’s related to my site recently dropping the index.php via htaccess. When I try to use trace=”true,” I get no indication that the plugin is opening the file. So then I tried to view a basic html or text file in the root directory and couldn’t get to them, either.

That’s why I’m wondering if htaccess is preventing the plugin from getting the csv file. Here is the htaccess:

RewriteEngine on RewriteCond $1 !^(images|(admin_dir)|themes|favicon.ico|robots.txt|index.php) [NC] RewriteRule ^(.*)$ /index.php/$1 [L]

Do you think that might be the problem or something else?

Well Scott, it doesn’t seem we have much luck here 😉

In addition to my previous question I also wondered if someone has encountered the following issue.

When importing the data the categories are created. They are all created at the same level and I move some of the categories below a parent category. Those become a child category so to speak. When I do another import with similar categories it will create the same category (which has been changed to child category) on the first level again. This means I have duplicate categories. One at first level and another at the second level. Luckely, the CP allows to change multiple entries to a different category in one go but with about 2000 products importing at the same time this can be quite time consuming.

Well, maybe someone has a solution to it.

regards, Tammo

fyi, my EE and browser were FREAKING out hanging last night and creating 1000’s of blank entries trying to run CSVGrab (which i’ve used, and loved, on other sites). i was so mad, i went to bed, woke up this morning, and changed the url to be a local server path. voila. i’m on engine hosting.

Is there a way to use the plugin to delete all entries not contained within the CSV? I get that it imports, and that the “unique” parameter allows for some entries to be overwritten with changed ones. But is there a parameter that deletes all entries left out of the CSV? Or maybe a new feature? *cough

(I’ve searched the usage but haven’t come up with how to do this - my apologies if I’ve missed it.)

EDIT:: I suppose marking it closed would produce the same effect. Does the plugin keep track of which records it adds/edits? All the rest could subsequently be marked as closed.

Thanks for your input

One additional question - I need to overwrite entries that are not considered “Unique”, not skip them - does this do that automatically?

Is there a way to use the plugin to delete all entries not contained within the CSV?

Not via the plugin, but in the EE control panel, you could delete all the entries in the weblog before you import the CSV, so that it imports into a clean, empty weblog. I do this all the time while testing.

Is there a way to use the plugin to delete all entries not contained within the CSV?

Not via the plugin, but in the EE control panel, you could delete all the entries in the weblog before you import the CSV, so that it imports into a clean, empty weblog. I do this all the time while testing.

Details? I can’t find this anywhere in the docs. Thanks.

I wrote a hack to escape all unique restrictions for a particular client we have where: ~every row in their CSV files is unique, though no one field can be guaranteed as an index ~the entire csv file contains all up-to-date info that should exist in the blog at any one time.

Its pretty simple and has probably been done before, yet its not for anyone who does not delete all prior entries before importing on a csv or else it is not for anyone who will be importing on a csv that still has records of entries that have already been imported. line 559:

} elseif ($unique == "none"){

return true;
        } else {

            /* Build custom query */

line 564

Where you put “none” in the unique= field.

Great Plugin Andrew,

I just like to make a feature request for CSVGrab to support FieldFrame Matrix. As of now, importing data is requiring data to be parsed using serialize() in PHP.

For example to get this

'test post'|'Lorem ipsum dolor sit amet'|'a:2:{i:0;a:2:{i:1;s:22:"Image - Soaring_Higher";i:2;a:1:{s:9:"file_name";s:18:"Soaring_Higher.jpg";}}i:1;a:2:{i:1;s:29:"Image - Orca_Whale_Underwater";i:2;a:1:{s:9:"file_name";s:25:"Orca_Whale_Underwater.jpg";}}}'

I have to run a similar code e.g.

<h1>partial data</h1>
<?
$array_1 = array(
    array('1' => 'Image - Soaring_Higher', array('file_name' => 'Soaring_Higher.jpg')), 
    array('1' => 'Image - Orca_Whale_Underwater', array('file_name' => 'Orca_Whale_Underwater.jpg')),
);

echo serialize($array_1) . '
';
echo '<pre>';
print_r ($array_1);
echo '</pre>

<p>’;
?>

I’m having trouble getting CSVGrab to work.

When i run the script (http://new.verdel.it/index.php/search/update):

START
Checking:
Found new entry:
Checking:
Checking:
Checking:
Checking:
Checking:
Checking:
Checking:
Checking:
Checking:
Checking:
Checking:
... etc ...

In this case there is an entry added with no value’s added (no title, etc.). If I run it a second time, there isn’t a second entry added.

My executed code:

START
{exp:csvgrab url="/files/quote.csv"
delimiter="TAB"
encloser=""
weblog="2"
skip="1"

title="22"

use="1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25"
fields="obr|artikelnummer|staffelminimum|ontvangstdatum|prijs_inclBTW|combi_omschrijving|unspsc|prijsverkoop|ondergroep|groepomschrijving|artnrfabrikant|verpakkingseenheid|EAN|productgroepnr|merk|voorraad|URLfoto|prijsverkoop_inclBTW|prijsdealer|hoofdgroep|afbeelding|artikelomschrijving|prijsdealer2|prijs2_inclBTW|staffel2"
unique="artikelnummer"
}
STOP

Even when I limit the ‘use’ and ‘fields’ to the bare minimum, I still get the same result.

If use a full url for the .csv file (http://new.verdel.it/files/quote.csv) i only get:

START STOP

Does anyone have an idea, i do need to get this to work:)

Try using the full server path in the URL, e.g. /home/12345/domains/mydomain. I know with some hosts they don’t allow you to connect to the typical website/file address through CSVGrab; this has happened to me before.

To find your path, the Movable Type website has a short tutorial at http://www.sixapart.com/movabletype/kb/installation/how_can_i_find.html

If I use my full server path i have the following code:

START

{exp:csvgrab url="/var/www/vhosts/new.verdel.it/httpdocs/files/quote.csv"
delimiter="TAB"
encloser=""
weblog="2"
skip="1"

title="22"

use="1|2|3|4|5|6|7|8|9|10|11|12|13|14|15|16|17|18|19|20|21|22|23|24|25"
fields="obr|artikelnummer|staffelminimum|ontvangstdatum|prijs_inclBTW|combi_omschrijving|unspsc|prijsverkoop|ondergroep|groepomschrijving|artnrfabrikant|verpakkingseenheid|EAN|productgroepnr|merk|voorraad|URLfoto|prijsverkoop_inclBTW|prijsdealer|hoofdgroep|afbeelding|artikelomschrijving|prijsdealer2|prijs2_inclBTW|staffel2"
unique="artikelnummer"
}

STOP

This outputs:

START STOP

So that doesn’t work either.

If i enter a gibrish external url like url=”http://new.vesdfrdel.iasft/files/quote.csv” i get:

Checking:
Checking:
Checking:
Checking:
Checking:
Checking:

Same output as: url=”/files/quote.csv” This is also the case for when I enter a interal gibrish url like url=”/filesasdfquote.csv”

I guess that when it cant read the file it outputs ‘Checking: …’ and if I enter a valid url like /var/www/vhosts/new.verdel.it/httpdocs/files/quote.csv or http://new.verdel.it/files/quote.csv it just isn’t processing anything.

Any ideas?

Have you tried limiting the use= and field= parameters to a single unit? That way you can debug any input errors with the script.

Is there a way to use the plugin to delete all entries not contained within the CSV?

Not via the plugin, but in the EE control panel, you could delete all the entries in the weblog before you import the CSV, so that it imports into a clean, empty weblog. I do this all the time while testing.

Details? I can’t find this anywhere in the docs. Thanks.

In the admin, go to Edit and select the Weblog you want to view. Then change the amount of results to 100, assuming you have a high number of entries, then Search. When the screen refreshes, select all of the entries with the checkbox and choose Delete.