Plugin: CSVGrab

Hi, you have

enclosure="QUOTE"

. Try

encloser="QUOTE"

.

Thanks a bunch. That was it.

I am importing data from an old website into a new EE version. We have 10 years worth of monthly magazines we are publishing on line. Each month has a monthly feature, editorial and a topic for each day. I am using 2 weblogs: monthly and daily. I am able to import the data into them just fine (did a limited test). However, I am not sure how to handle the relationship field.

Each daily is related to the month’s issue it was published in. How would I import the daily data so that it automatically is related to a particular month? I would assume I would have a column that I input the related field data…but what data is that? The entry id for the month?

Found this and it is the same kind of issue, but they never did say how they got around it: http://ellislab.com/forums/viewthread/43857/P18/

Hi,

I’ve been using CSVGrab few times and had no problem. I just tried running a small import for 50 lines of CSV file and it is not working.

I turned on the “TRACE” option and it returns “TRACE: Resource id #45” error.

My CSV file is using UTF8 encoding and Unix (\n) line break option.

Any help would be appreciated.

Regards Rudy

Hi - the “TRACE: Resource id #45” is not an error: it indicates that the file is being opened. However, if you see nothing after that then CSVgrab is having trouble parsing the file or matching it up with your template, or there’s an error in the template.

Iff the “use” and “fields” value in your template are not in one-to-one correspondence, then CSVgrab will fail silently.

If you post your template and some sample data lines, that could help.

Dan

I need some help here.

I tried to use CSVgrab to add entries with multiple categories using the category IDs rather than the category name. The entries were added to the weblog, but the categories were not assigned to the entries.

I’m using CSVGrab v.0.2.1

And here’s the template I’m using:

{exp:csvgrab url="http://www.domain.com/import/sample_feb14.csv"
   delimiter=","
   encloser="QUOTE"
   skip="1"
   date="8"
   weblog="3"
   title="6"
   category_field="3"
   use="4|7"
   fields="submitted_url|submitted_summary"
   unique="title,submitted_url"
}

And here’s a sample row of csv data:

36,"test","26,58,60","http://www.url.com","nickname","title","summary","2010-01-23 18:00:00"

The category field contains the category IDs of three existing categories. Can anyone tell me why the categories were not assigned to the entries?

And there’s another issue: based on the output of the template, it seemed that CSVgrab tried to create three new categories named 25, 58, and 60. And when I looked at the exp_categories table, sure enough, there are three new categories with names (not IDs) 25, 58 and 60. And the group ID for these unwanted categories is 0. Also, these three categories and the category group zero are not visible anywhere in “Admin › Weblog Administration › Category Groups” in the control panel. Can I safely delete these three records from the exp_categories table?

I think you’ll need to use the category names, rather than the ids in you CSV file.

There is a category_group= parameter to choose which group new categories are created in. eg, category_group=”1”

Andrew

Thanks for your reply.

PHP is not my strong point, but looking at the code of the plugin, it looks like we should be able to used category IDs.

Lines 384 - 391:

if (isset($this->cats) && $this->cats != '') 
    {            
         $cats = explode(',',$this->cats);
         $cats = array_unique($cats);

         $results = $DB->query("SELECT cat_id, parent_id FROM exp_categories 
        WHERE (cat_id IN ('".implode("','",$cats)."') OR cat_name IN ('".implode("','",$cats)."'))
        AND group_id = '".$query->row['cat_group']."'");

And for those unwanted categories in the exp_categories table, can I delete them manually without damaging anything?

I need to import old entry dates and keep them that way. When I use date= and the number of the field, it seems to work fine. But when I look at the entry in EE, I see today’s date as the entry date and not the old date.

Am I misunderstanding how the date parameter works? Is there a way of keeping an old entry date?

The date format appears to be consistent with how it is described on the csvgrab Web site.

Hi all,

I am currently testing a new module for EE 2.0 to replace the FeedGrab, CSVGrab and XMLGrab plugins.

For more details, and if you’d like to help test it, please visit:

DataGrab for ExpressionEngine 2.0

Thanks, Andrew

BTW - If anyone’s trying to get this to work with gypsy fields, change the query to this:

$customfieldsArray = $DB->query("SELECT exp_weblog_fields.field_id, 
                                exp_weblog_fields.field_name, 
                                exp_weblog_fields.field_label, exp_weblog_fields.field_fmt
                                 FROM exp_weblogs, exp_weblog_fields
                                 WHERE exp_weblogs.field_group = exp_weblog_fields.group_id
                                 OR field_is_gypsy = 'y'
                AND gypsy_weblogs
                LIKE '%".$this->feed_weblog_id."%' OR exp_weblogs.weblog_id = '".$this->feed_weblog_id."'");

I love CSV Grab and have used it a number of times to quickly and easily import loads of data.

Is there a way to append the content that has already been uploaded through CSV Grab?

For example, if I import 80 rows of data with 10 cells per row and then a week later the content of the .csv file is updated, modifying some content and adding new, is it possible to re-import that .csv file and have it append the entries on the site? I tried to do it and the way I currently have it configured it simply creates duplicate entries.

Current configuration:

{exp:csvgrab url="http://www.mysite.edu/projects/list.csv"
         weblog="18"
         title="1"
         skip="2"
         delimiter=","
         author="21"
         encloser="QUOTE"
         use="1|2|3|4|5|6|7|8|9"
         fields="research-project-title|research-topic|research-faculty-member|research-project-sponsor|research-project-start-date|research-project-end-date|research-location|research-project-partners|research-project-description" }

Thanks in advance for the assistance on this.

Forrest

Am trying to work out if CSVGrab will do what I need for an upcoming project:

1) Can I import automatically, say, every 24 hours? I’m going to have a new CSV file uploaded nightly to the server so need to just update the appropriate weblog each night.

2) Can this process overwrite all the old weblog entries, or will it cause duplicates? (Only some of the data is going to change each night but I’ll only have the full CSV to work with each time.)

3) I understand that CSVGrab can set the category for entries from the CSV. How does this work with sub-categories? For, say, areas of the world, if I want an entry to be categorised as both “Asia” and “Japan”, can the CSV just have “Asia|Japan” for that entry and I take the categories from that? Will EE understand that Japan is a sub-category of Asia? How would I create new sub-categories?

Thanks loads for your help

Hi,

I just thought I would post my experience here in using the csvgrab.

My primary intention was to get the csvgrab to import and setup a large category group list for me. It worked perfectly !!! 😊

One major hiccup I had which is worth nothing is that I was working offline while developing my site, and this made the csvgrab useless, so you will need to turn your System to ‘On’ online to get the csvgrab process to work.

And another finicky thing that caused me problems, believe it or not, was the curly quotes used within the {exp:csvgrab} parameters. I was copying directly from the plugin documentation within my EE Control Panel, and I noticed (after almost screaming) that the curly quotes were different when I cut and paste from various sources, so please make sure that you physically type the double quotes, or make sure they are consistent within the {exp:csvgrab} tags and parameters.

Basically then, I followed Andrew Weaver’s csvgrab plugin instruction on setting up a ‘dummy weblog’ etc with custom_fields and my ultimate desired Category Group.

I then set up an excel / csv file with 4 columns and only one Row Entry.

So in the first column cell, I placed all my categories separated by a ‘comma’. This is the crucial part here. You don’t put the categories in individual rows or cells. Put them all in one cell as the csvgrab process then reads all the categories associated with this one entry.

In the second column cell I placed a dummy title

In the third column cell I placed my dummy custom_field entry

In the fourth column cell I placed my dummy custom_field entry

I then followed Andrew Weaver’s instructions in setting up the ‘go’ template AND basically it worked well by inserting one entry and then inserting all the categories in my Category Group.

You just need to be thinking clearly on whether your categories have spaces within one category name, and then the delimiters you use. Note here below that I used the semi-colon as the csv column delimiter and then the comma as the actual category delimiter which defined my individual categories.

{exp:csvgrab url="http://www.website.com/mycsvfile.csv"
    delimiter=";"
    encloser="QUOTE"
    weblog="13"
    category_group="19"
    category_field="1"
    category_delimiter=","
    title="2"
    use="3|4"
    fields="custom_field|custom_field"
}

Hope this helps someone else who has struggled with the notion of having to hand-type hundreds of categories. 😊

Thank you Andrew for producing a brilliant plugin.

okay, that’s awesome. i’ll file that in the back of my tiny head for future ref.