Plugin: CSVGrab

Hi Dan,

Thanks for your reply. I read through the whole post but there are so much comments I don’t know where to start. Also, most posts have at least an error message (I wish I had one) and those posts whith no error (like me) never explained the solution. I did added the trace to the template but I get no output. I also tried text file and csv. Loaded in ASCI. btw, I am on a windows XP pc. Tried home path and full url, not any result at all.

Here is my scv file CSV file

I did generate the csv file from excell, maybe I try that other tool mentioned earlier in the post.

Cheers, Tammo

Just noticed the quote in the skip line and thought it looked akward and typed it again and rendered the template…and finally I am getting somewhere…or at least, I have got an error and trace output. But at least that gives a lead how to solve it.

error:

TRACE: Array ( [title] => Array ( [is_custom] => 0 [field] => 1 ) [date] => Array ( [is_custom] => 0 [field] => ) [website] => Array ( [is_custom] => 1 [field] => 2 [id] => 10 [format] => none ) [description] => Array ( [is_custom] => 1 [field] => 3 [id] => 11 [format] => xhtml ) [phone] => Array ( [id] => 12 [format] => none ) [email_address] => Array ( [id] => 13 [format] => none ) [directory_country] => Array ( [id] => 37 [format] => none ) [directory_city] => Array ( [id] => 38 [format] => xhtml ) [address] => Array ( [id] => 55 [format] => xhtml ) [province] => Array ( [id] => 56 [format] => none ) [latitude] => Array ( [id] => 53 [format] => xhtml ) [longitude] => Array ( [id] => 54 [format] => xhtml ) [zip] => Array ( [id] => 57 [format] => xhtml ) [end_event_date] => Array ( [id] => 65 [format] => xhtml ) [start_event_date] => Array ( [id] => 64 [format] => none ) ) TRACE: Resource id #30 TRACE: 2 - Array ( [0] => WineJobs.com [1] => WineJobs.com [2] => http://winejobs.com [3] => The leading online wine industry job board. ) Fatal error: Call to undefined function: mb_convert_encoding() in /home/system/public_html/hospisys/plugins/pi.csvgrab.php on line 231

I read that Ingmar thinks it indicates missing Unicode support (mb = multibyte) on the host so I will check that out.

Glad you are getting somewhere. I saw the bad quote but thought that was an artifact of the pretty-printer. I read back further and I see that it might be in Andrew’s example.

The mb_convert_encoding() is one problem. But also add encloser=”QUOTE” to your attributes if you have further problems. CSV’s produced by Excel don’t have them unless necessary. But fgetcsv(), as used by CSVGrab, may fail if that parameter is not specified. However, it was not required in older versions of PHP, which is why maybe you didn’t see that error.

To figure out what version of PHP you have, and what support it was built with, make up a simple phpinfo.php page with this:

<?php phpinfo(); ?>

and fetch it. It will dump all kinds of useful info.

thanks. I have PHP version 4.4.7. I looked at the phpinfo file but can’t figure out where it is stated if unicode is supported but I guess it is as FeedGrab works fine for me. The mb_convert_encoding issue was also discussed in this post where Andrew came up with a solution. So I hope there is a solution already available for CSVGrab.

Everything is working great for me but the author piece. I have used 1, 2, and 3 as inputs and nothing changes when I open the article. They are all entered in by the user associated to 1. What am I missing?

Ok, I deleted

// Get unique url
            $source_encoding = ( isset( $this->RSS->source_encoding ) ? $this->RSS->source_encoding : 'UTF-8' );
            $encoded_title = mb_convert_encoding($this->post[ "title" ], strtoupper('ISO-8859-1'), $source_encoding ); // Fixes warning on next line
            $this->post[ "url_title" ] = $REGX->create_url_title( $encoded_title );
            $sql = "SELECT count(*) AS count FROM exp_weblog_titles WHERE url_title = '".$DB->escape_str( $this->post[ "url_title" ] )."' AND weblog_id = '$this->feed_weblog_id'";
            $results = $DB->query($sql);
            if ($results->row['count'] > 0) {
                // Duplicate exists, add a counter on the end
                $sql = "SELECT count(*) AS count FROM exp_weblog_titles WHERE url_title LIKE '".$DB->escape_str( $this->post[ "url_title" ] )."%' AND weblog_id = '$this->feed_weblog_id'";
                $results = $DB->query($sql);
                $this->post[ "url_title" ] .= $results->row['count']+1;
            }

now it works except for the title url which stays empty. Also the following notice shows after rendering:

Notice: Undefined index: url_title in /home/hospital/public_html/hospisys/plugins/pi.csvgrab.php on line 278

But at least the entry is created, hopefully someone can help me with the title_url.

Regards, Tammo

I cannot get this to work at all. I’m on a mac, using Transmit and have followed every hint/tip/trick I’ve found in this thread.

I can’t even get the template to load because I continue to get this message.

Safari can’t open the page “http://daktech.philleitch.com/index.php/inc/go”. The error was: “cannot decode raw data” (NSURLErrorDomain:-1015) Please choose Report Bugs to Apple from the Safari menu, note the error number, and describe what you did before you saw this message

My template

{exp:csvgrab url="http://daktech.philleitch.com/daktech_parts.csv" 
weblog="37" 
title="1"
delimiter=","
stagger="3600"
use="2|3|4|5" 
fields="parts_part_number|parts_description|parts_cost|parts_list_price" }

I’ve attached my csv file.

Any help at all would be greatly appreciated.

Phil,

I think there are two unrelated problems here.

Try adding encloser=”QUOTE” to your template. Also add trace=”true”.

However, the Safari error you are seeing is a webserver problem. I see an equivalent error when using Firefox. IE and Chrome also fail to load the page, and give less informative errors. This error is happening because your browser says it is willing to accept compressed data from the server (which saves bandwidth). Most all browsers say they will. But the compressed output coming back is corrupted in some way, and the browser can’t uncompress it.

I Googled this, and I see that it may be due to a cache with a corrupted entry that’s in front of the webserver. But it would have to be at the webhost, not on your side, because I see that error too. It might also be to a corrupted Zend cache. In any case, this is something you should bring up with your webhosting company.

One more note: your .csv file has the right kind of line endings, so that’s not a problem. But it also has some non-ASCII characters in the line about “USB Docking Station”.

Thanks Dan. I’ve contacted my webhost (Dreamhost, erf!) and hopefully this has put me on the right path. I had a feeling that it wasn’t the plug-in or me because the error was something I couldn’t find either here on the EE forums or anywhere else for that matter.

Glad there are smarter people than me here to help me out.

Again, thanks and I will update if/when I get this fixed in case anybody else ever has the same problem.

UPDATE: Well, adding encloser=”QUOTE” and trace=”true” along with removing the non-ASCII characters seems to have done the trick. I am officially feeling stupid.

Thanks Dan for setting me straight.

Hi, just to let know my host has installed the mb unicode support extension and now it works fine. Thanks for the help.

regards, Tammo

Hello, I installed the plug-in (version 0.2.1) on EE 1.6.6 (licence version) and tried to import a very basic csv file, just to test the plug-in. Here’s the csv file:

"1"_"mycategory"_"my title 3"_"2"_"2007-02-17"_"Some dummy text"
"2"_"mycategory"_"my title 4"_"2"_"2008-01-01"_"Some other dummy text"

where the first field is not used at the moment, “mycategory” is an existing category, “2” is the author ID , “my title” is obviously the entry title and “some dummy text” is the text to be imported in the body field; the date is formatted as specified in the documentation.

My template looks like that:

{exp:csvgrab url="http://www.artearti.net/artesys/sample_csv.csv"
    delimiter="_"
    encloser="QUOTE" 
    weblog="1"
    author="2”
    date="5"
    category_group="1"
    title="3"
    category_field="2"
    use="6"
    fields="body"
    trace="true"
}

When running the “go” template, everything seems to work, no errors and no warnings, 2 entries imported. However when I get back to the CP I cannot see the entries listed in the “Edit” screen, even if the entries are regularly published on the website. I can only see the new entries in the CP home page. When I get to the edit screen, I can see that 2 of the csv values are imported regularly (title, body) while date is set to today and the author is NOT imported, causing EE to generate an error when loading the publish page. If I manually add the author everything’s fine and the entry gets listed in the edit screen. I assume I’m doing something wrong, but I cannot see the mistake. Any suggestion would be much appreciated, ‘cause I need to import at least 600 entries (all long articles) and I don’t want to do that manually.

Partially solved: I tried to remove the extra field in the CSV and update the template. Now the plug-in grabs the author but the date is still set to today. Any idea?

lonelytraveller,

Not sure if this is the problem, but the quote after the 2 in author is not a regular ASCII double quote, but a curly quote. I believe one of Andrew’s examples has a bad quote in it. If that is still the case in your template, try fixing that and see what happens.

Wanted to check on the hopeful prospect of including related entries in the upload. Please advise.

Thanks to Dan for the suggestion and thanks to Andy who PM me suggesting to include all the fields I wanted to import except title in the “use” and “fields” parameters.

However none of the above solutions fixed the problem: I still cannot import the date. I switched the “Trace” function on, but it doesn’t seem to detect any problem.

I finally imported everything through PhpMyAdmin directly into the DB, a couple of queries did the rest of the job. I only had to convert the date to Unix timestamp format.

However I want to report a couple of problems I had while still testing the plug-in, since I think this info could be useful to both developers and users:

• The plug-in imported some entries without any title, because in one of the steps I chose the wrong colum for the title field. The entries could be visible on the front-end, ‘cause EE automatically created a url (assigned a progressive number) and an ID, but the entries could not be modified in the back-end. However when deleted those entries could still be visible both in the front-end and in the DB tables (just in case someone has the same problem, I had to fix it in the DB, working on both weblog_titles and weblog_data tables).

• Some other entries where imported without the weblog id, which is quite strange, ‘cause all the other fields where imported without problems. I had only one weblog (id=1). However until I assigned the entries to the weblog (manually updating the db) I couldn’t make any modification to the entry;

• finally, as someone else pointed-out in this thread, one of the import duplicated some categories, and even created a blank one. Again I tried to delete them from the CP, but they still showed up in the front-end. This problem too had to be fixed working on the DB.

Is there a way to prevent such errors? I mean, if the user puts the wrong info in the “go” template (as I did), is there a way to prevent duplicated categories or entries imported without some key-field like title and weblog id? An entry without any information in these fields cannot be handled from the CP.

I hope this info can be useful to someone else.